[jira] [Commented] (YARN-4734) Merge branch:YARN-3368 to trunk
[
https://issues.apache.org/jira/browse/YARN-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440554#comment-15440554
]
Hadoop QA commented on YARN-4734:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m 4s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 5s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 1s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 39s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
11s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 34s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 30s
{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 33s
{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m 13s
{color} | {color:red} The patch generated 46 new + 75 unchanged - 0 fixed = 121
total (was 75) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 41 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 9s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 5m 28s
{color} | {color:red} root generated 6 new + 11318 unchanged - 0 fixed = 11324
total (was 11318) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 119m 36s
[jira] [Commented] (YARN-4734) Merge branch:YARN-3368 to trunk
[
https://issues.apache.org/jira/browse/YARN-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440423#comment-15440423
]
Hadoop QA commented on YARN-4734:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 6s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 37s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
12s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 32s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 30s
{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 26s
{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m 13s
{color} | {color:red} The patch generated 46 new + 75 unchanged - 0 fixed = 121
total (was 75) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 41 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 9s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 5m 2s
{color} | {color:red} root generated 6 new + 11318 unchanged - 0 fixed = 11324
total (was 11318) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 55s
[jira] [Created] (YARN-5570) NodeManager blocks kill attempts during startup if ResourceManager is down
Andrew Wang created YARN-5570:
-
Summary: NodeManager blocks kill attempts during startup if
ResourceManager is down
Key: YARN-5570
URL: https://issues.apache.org/jira/browse/YARN-5570
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.7.3
Reporter: Andrew Wang
Found while testing a 2.7.3 RC:
{quote}
* If you start the NodeManager and not the RM, the NM has a handler for SIGTERM
and SIGINT that blocked my Ctrl-C and kill attempts during startup. I had to
kill -9 it.
{quote}
[~ste...@apache.org] tells me that this is potentially addressed by YARN-679,
though that's a large patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5569) YARN logs out user when job is run when improperly configured
Andrew Wang created YARN-5569:
-
Summary: YARN logs out user when job is run when improperly
configured
Key: YARN-5569
URL: https://issues.apache.org/jira/browse/YARN-5569
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.7.3
Reporter: Andrew Wang
This is something that Vinay and I both hit while testing a 2.7.3 RC:
{quote}
I didn't have SSH to localhost set up (new laptop), and when I tried to run the
Pi job, it'd exit my window manager session. I feel there must be a more
developer-friendly solution here.
{quote}
{quote}
Faced same issues as Andrew wang, while running the WordCount job first time in
my new Ubuntu installation, without 'configuring the shuffle handler properly'.
Whole session got logged by closing all other applications open. After
configuring the shuffle handler properly, job was successful though.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5567) NodeHealthScriptRunner#reportHealthStatus bug fix
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440361#comment-15440361
]
Hadoop QA commented on YARN-5567:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 6s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 49s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12825732/YARN-5567.001.patch |
| JIRA Issue | YARN-5567 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8a585aedcbae 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 19c743c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12911/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12911/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> NodeHealthScriptRunner#reportHealthStatus bug fix
> -
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5567.001.patch
>
>
> In case of FAILED_WITH_EXIT_CODE, health status should be false.
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
>
[jira] [Commented] (YARN-4889) Changes in AMRMClient for identifying resource-requests explicitly
[ https://issues.apache.org/jira/browse/YARN-4889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440329#comment-15440329 ] Hudson commented on YARN-4889: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10362 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10362/]) YARN-4889. Changes in AMRMClient for identifying resource-requests (wangda: rev 19c743c1bbcaf3df8f1d63e557143c960a538c42) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestNMClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/RemoteRequestsTable.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientContainerRequest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestDistributedScheduling.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java > Changes in AMRMClient for identifying resource-requests explicitly > -- > > Key: YARN-4889 > URL: https://issues.apache.org/jira/browse/YARN-4889 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Arun Suresh > Fix For: 2.9.0 > > Attachments: YARN-4889.001.patch, YARN-4889.002.patch, > YARN-4889.003.patch, YARN-4889.004.patch > > > YARN-4879 proposes the notion of identifying allocate requests explicitly.. > This JIRA is to track the changes in AMRMClient to keep it wire compatible > with the changes. Please refer to the design doc in the parent JIRA for > details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5327) API changes required to support recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440303#comment-15440303 ] Hudson commented on YARN-5327: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10361 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10361/]) YARN-5327. API changes required to support recurring reservations in the (subu: rev b930dc3ec06afa479a249490976e3e127d201706) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestReservationInputValidator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/ReservationInputValidator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ReservationDefinition.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ReservationDefinitionPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto > API changes required to support recurring reservations in the YARN > ReservationSystem > > > Key: YARN-5327 > URL: https://issues.apache.org/jira/browse/YARN-5327 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Fix For: 2.9.0 > > Attachments: YARN-5327.001.patch, YARN-5327.002.patch, > YARN-5327.003.patch, YARN-5327.004.patch, YARN-5327.005.patch, > YARN-5327.006.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to track the changes needed > in ApplicationClientProtocol to accomplish it. Please refer to the design doc > in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5327) API changes required to support recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440284#comment-15440284 ] Sangeetha Abdu Jyothi commented on YARN-5327: - Thanks [~subru]! :) > API changes required to support recurring reservations in the YARN > ReservationSystem > > > Key: YARN-5327 > URL: https://issues.apache.org/jira/browse/YARN-5327 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Fix For: 2.9.0 > > Attachments: YARN-5327.001.patch, YARN-5327.002.patch, > YARN-5327.003.patch, YARN-5327.004.patch, YARN-5327.005.patch, > YARN-5327.006.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to track the changes needed > in ApplicationClientProtocol to accomplish it. Please refer to the design doc > in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5567) NodeHealthScriptRunner#reportHealthStatus bug fix
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440232#comment-15440232
]
Ray Chiang commented on YARN-5567:
--
Thanks [~yufeigu] for your contribution. This looks like a straightforward fix.
> NodeHealthScriptRunner#reportHealthStatus bug fix
> -
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5567.001.patch
>
>
> In case of FAILED_WITH_EXIT_CODE, health status should be false.
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4734) Merge branch:YARN-3368 to trunk
[ https://issues.apache.org/jira/browse/YARN-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440226#comment-15440226 ] Hadoop QA commented on YARN-4734: - (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-YARN-Build/12910/console in case of problems. > Merge branch:YARN-3368 to trunk > --- > > Key: YARN-4734 > URL: https://issues.apache.org/jira/browse/YARN-4734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-4734.1.patch, YARN-4734.2.patch, YARN-4734.3.patch, > YARN-4734.4.patch, YARN-4734.5.patch, YARN-4734.6.patch, YARN-4734.7.patch, > YARN-4734.8.patch, YARN-4734.9-NOT_READY.patch > > > YARN-2928 branch is planned to merge back to trunk shortly, it depends on > changes of YARN-3368. This JIRA is to track the merging task. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4734) Merge branch:YARN-3368 to trunk
[ https://issues.apache.org/jira/browse/YARN-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440221#comment-15440221 ] Hadoop QA commented on YARN-4734: - (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-YARN-Build/12909/console in case of problems. > Merge branch:YARN-3368 to trunk > --- > > Key: YARN-4734 > URL: https://issues.apache.org/jira/browse/YARN-4734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-4734.1.patch, YARN-4734.2.patch, YARN-4734.3.patch, > YARN-4734.4.patch, YARN-4734.5.patch, YARN-4734.6.patch, YARN-4734.7.patch, > YARN-4734.8.patch, YARN-4734.9-NOT_READY.patch > > > YARN-2928 branch is planned to merge back to trunk shortly, it depends on > changes of YARN-3368. This JIRA is to track the merging task. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4734) Merge branch:YARN-3368 to trunk
[ https://issues.apache.org/jira/browse/YARN-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4734: - Attachment: YARN-4734.9-NOT_READY.patch Attached patch to trigger Jenkins. (Not ready for review yet) > Merge branch:YARN-3368 to trunk > --- > > Key: YARN-4734 > URL: https://issues.apache.org/jira/browse/YARN-4734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-4734.1.patch, YARN-4734.2.patch, YARN-4734.3.patch, > YARN-4734.4.patch, YARN-4734.5.patch, YARN-4734.6.patch, YARN-4734.7.patch, > YARN-4734.8.patch, YARN-4734.9-NOT_READY.patch > > > YARN-2928 branch is planned to merge back to trunk shortly, it depends on > changes of YARN-3368. This JIRA is to track the merging task. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
[ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440197#comment-15440197 ] Wangda Tan commented on YARN-4945: -- Thanks for discussion, [~eepayne]. [~sunilg]. I think normalization for inter-queue / intra-queue preemption is one of the top priority goal for this feature. If you take a look at existing preemption code, it normalizes preempt-able resource for reserved-container-candidate-selector and fifo-candidate-selector. We can do the similar normalization for inter/intra-queue preemption. Once [~sunilg] has the patch with more test cases ready, we can do some reviews / tests to make sure it works. > [Umbrella] Capacity Scheduler Preemption Within a queue > --- > > Key: YARN-4945 > URL: https://issues.apache.org/jira/browse/YARN-4945 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan > Attachments: Intra-Queue Preemption Use Cases.pdf, > IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.patch > > > This is umbrella ticket to track efforts of preemption within a queue to > support features like: > YARN-2009. YARN-2113. YARN-4781. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5327) API changes required to support recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440155#comment-15440155 ] Wangda Tan commented on YARN-5327: -- Looks good, +0, thanks for update! > API changes required to support recurring reservations in the YARN > ReservationSystem > > > Key: YARN-5327 > URL: https://issues.apache.org/jira/browse/YARN-5327 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Attachments: YARN-5327.001.patch, YARN-5327.002.patch, > YARN-5327.003.patch, YARN-5327.004.patch, YARN-5327.005.patch, > YARN-5327.006.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to track the changes needed > in ApplicationClientProtocol to accomplish it. Please refer to the design doc > in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4889) Changes in AMRMClient for identifying resource-requests explicitly
[ https://issues.apache.org/jira/browse/YARN-4889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440157#comment-15440157 ] Arun Suresh commented on YARN-4889: --- Thanks for the reviews [~leftnoteasy], [~kasha] and [~subru] ... > Changes in AMRMClient for identifying resource-requests explicitly > -- > > Key: YARN-4889 > URL: https://issues.apache.org/jira/browse/YARN-4889 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Arun Suresh > Attachments: YARN-4889.001.patch, YARN-4889.002.patch, > YARN-4889.003.patch, YARN-4889.004.patch > > > YARN-4879 proposes the notion of identifying allocate requests explicitly.. > This JIRA is to track the changes in AMRMClient to keep it wire compatible > with the changes. Please refer to the design doc in the parent JIRA for > details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5568) Fix issues Rumen to SLS converter
Wangda Tan created YARN-5568: Summary: Fix issues Rumen to SLS converter Key: YARN-5568 URL: https://issues.apache.org/jira/browse/YARN-5568 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan Found some issues while trying to convert Rumen data to SLS data, for example, some int -> long converting issue, and script issue, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5202) Dynamic Overcommit of Node Resources - POC
[
https://issues.apache.org/jira/browse/YARN-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440131#comment-15440131
]
Nathan Roberts commented on YARN-5202:
--
Yahoo has received a letter accusing the patches originally contributed as part
of this jira (specifically YARN-5202.patch dated 06/Jun/16 16:35, and
YARN-5202-branch2.7-uber.patch dated 18/Jul/16 12:45) of infringing two U.S.
patents. While Yahoo believes this assertion has no merit, the patches have
been removed pending further investigation.
> Dynamic Overcommit of Node Resources - POC
> --
>
> Key: YARN-5202
> URL: https://issues.apache.org/jira/browse/YARN-5202
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Nathan Roberts
>Assignee: Nathan Roberts
>
> This Jira is to present a proof-of-concept implementation (collaboration
> between [~jlowe] and myself) of a dynamic over-commit implementation in YARN.
> The type of over-commit implemented in this jira is similar to but not as
> full-featured as what's being implemented via YARN-1011. YARN-1011 is where
> we see ourselves heading but we needed something quick and completely
> transparent so that we could test it at scale with our varying workloads
> (mainly MapReduce, Spark, and Tez). Doing so has shed some light on how much
> additional capacity we can achieve with over-commit approaches, and has
> fleshed out some of the problems these approaches will face.
> Primary design goals:
> - Avoid changing protocols, application frameworks, or core scheduler logic,
> - simply adjust individual nodes' available resources based on current node
> utilization and then let scheduler do what it normally does
> - Over-commit slowly, pull back aggressively - If things are looking good and
> there is demand, slowly add resource. If memory starts to look over-utilized,
> aggressively reduce the amount of over-commit.
> - Make sure the nodes protect themselves - i.e. if memory utilization on a
> node gets too high, preempt something - preferably something from a
> preemptable queue
> A patch against trunk will be attached shortly. Some notes on the patch:
> - This feature was originally developed against something akin to 2.7. Since
> the patch is mainly to explain the approach, we didn't do any sort of testing
> against trunk except for basic build and basic unit tests
> - The key pieces of functionality are in {{SchedulerNode}},
> {{AbstractYarnScheduler}}, and {{NodeResourceMonitorImpl}}. The remainder of
> the patch is mainly UI, Config, Metrics, Tests, and some minor code
> duplication (e.g. to optimize node resource changes we treat an over-commit
> resource change differently than an updateNodeResource change - i.e.
> remove_node/add_node is just too expensive for the frequency of over-commit
> changes)
> - We only over-commit memory at this point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5202) Dynamic Overcommit of Node Resources - POC
[
https://issues.apache.org/jira/browse/YARN-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Roberts updated YARN-5202:
-
Attachment: (was: YARN-5202.patch)
> Dynamic Overcommit of Node Resources - POC
> --
>
> Key: YARN-5202
> URL: https://issues.apache.org/jira/browse/YARN-5202
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Nathan Roberts
>Assignee: Nathan Roberts
>
> This Jira is to present a proof-of-concept implementation (collaboration
> between [~jlowe] and myself) of a dynamic over-commit implementation in YARN.
> The type of over-commit implemented in this jira is similar to but not as
> full-featured as what's being implemented via YARN-1011. YARN-1011 is where
> we see ourselves heading but we needed something quick and completely
> transparent so that we could test it at scale with our varying workloads
> (mainly MapReduce, Spark, and Tez). Doing so has shed some light on how much
> additional capacity we can achieve with over-commit approaches, and has
> fleshed out some of the problems these approaches will face.
> Primary design goals:
> - Avoid changing protocols, application frameworks, or core scheduler logic,
> - simply adjust individual nodes' available resources based on current node
> utilization and then let scheduler do what it normally does
> - Over-commit slowly, pull back aggressively - If things are looking good and
> there is demand, slowly add resource. If memory starts to look over-utilized,
> aggressively reduce the amount of over-commit.
> - Make sure the nodes protect themselves - i.e. if memory utilization on a
> node gets too high, preempt something - preferably something from a
> preemptable queue
> A patch against trunk will be attached shortly. Some notes on the patch:
> - This feature was originally developed against something akin to 2.7. Since
> the patch is mainly to explain the approach, we didn't do any sort of testing
> against trunk except for basic build and basic unit tests
> - The key pieces of functionality are in {{SchedulerNode}},
> {{AbstractYarnScheduler}}, and {{NodeResourceMonitorImpl}}. The remainder of
> the patch is mainly UI, Config, Metrics, Tests, and some minor code
> duplication (e.g. to optimize node resource changes we treat an over-commit
> resource change differently than an updateNodeResource change - i.e.
> remove_node/add_node is just too expensive for the frequency of over-commit
> changes)
> - We only over-commit memory at this point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5202) Dynamic Overcommit of Node Resources - POC
[
https://issues.apache.org/jira/browse/YARN-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Roberts updated YARN-5202:
-
Attachment: (was: YARN-5202-branch2.7-uber.patch)
> Dynamic Overcommit of Node Resources - POC
> --
>
> Key: YARN-5202
> URL: https://issues.apache.org/jira/browse/YARN-5202
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Nathan Roberts
>Assignee: Nathan Roberts
>
> This Jira is to present a proof-of-concept implementation (collaboration
> between [~jlowe] and myself) of a dynamic over-commit implementation in YARN.
> The type of over-commit implemented in this jira is similar to but not as
> full-featured as what's being implemented via YARN-1011. YARN-1011 is where
> we see ourselves heading but we needed something quick and completely
> transparent so that we could test it at scale with our varying workloads
> (mainly MapReduce, Spark, and Tez). Doing so has shed some light on how much
> additional capacity we can achieve with over-commit approaches, and has
> fleshed out some of the problems these approaches will face.
> Primary design goals:
> - Avoid changing protocols, application frameworks, or core scheduler logic,
> - simply adjust individual nodes' available resources based on current node
> utilization and then let scheduler do what it normally does
> - Over-commit slowly, pull back aggressively - If things are looking good and
> there is demand, slowly add resource. If memory starts to look over-utilized,
> aggressively reduce the amount of over-commit.
> - Make sure the nodes protect themselves - i.e. if memory utilization on a
> node gets too high, preempt something - preferably something from a
> preemptable queue
> A patch against trunk will be attached shortly. Some notes on the patch:
> - This feature was originally developed against something akin to 2.7. Since
> the patch is mainly to explain the approach, we didn't do any sort of testing
> against trunk except for basic build and basic unit tests
> - The key pieces of functionality are in {{SchedulerNode}},
> {{AbstractYarnScheduler}}, and {{NodeResourceMonitorImpl}}. The remainder of
> the patch is mainly UI, Config, Metrics, Tests, and some minor code
> duplication (e.g. to optimize node resource changes we treat an over-commit
> resource change differently than an updateNodeResource change - i.e.
> remove_node/add_node is just too expensive for the frequency of over-commit
> changes)
> - We only over-commit memory at this point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[
https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440125#comment-15440125
]
Hadoop QA commented on YARN-5549:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m 20s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 45s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12825722/YARN-5549.004.patch |
| JIRA Issue | YARN-5549 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0d132e53d3fa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 407b519 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12908/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12908/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> AMLauncher.createAMContainerLaunchContext() should not log the command to be
> launched indiscriminately
> --
>
> Key: YARN-5549
> URL: https://issues.apache.org/jira/browse/YARN-5549
> Project: Hadoop YARN
> Issue Type: Bug
>
[jira] [Updated] (YARN-5567) NodeHealthScriptRunner#reportHealthStatus bug fix
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5567:
---
Affects Version/s: 3.0.0-alpha1
2.8.0
> NodeHealthScriptRunner#reportHealthStatus bug fix
> -
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5567.001.patch
>
>
> In case of FAILED_WITH_EXIT_CODE, health status should be false.
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5567) NodeHealthScriptRunner#reportHealthStatus bug fix
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5567:
---
Description:
In case of FAILED_WITH_EXIT_CODE, health status should be false.
{code}
case FAILED_WITH_EXIT_CODE:
setHealthStatus(true, "", now);
break;
{code}
should be
{code}
case FAILED_WITH_EXIT_CODE:
setHealthStatus(false, "", now);
break;
{code}
was:
{code}
case FAILED_WITH_EXIT_CODE:
setHealthStatus(true, "", now);
break;
{code}
should be
{code}
case FAILED_WITH_EXIT_CODE:
setHealthStatus(false, "", now);
break;
{code}
> NodeHealthScriptRunner#reportHealthStatus bug fix
> -
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5567.001.patch
>
>
> In case of FAILED_WITH_EXIT_CODE, health status should be false.
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5567) NodeHealthScriptRunner#reportHealthStatus bug fix
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5567:
---
Attachment: YARN-5567.001.patch
> NodeHealthScriptRunner#reportHealthStatus bug fix
> -
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5567.001.patch
>
>
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5567) NodeHealthScriptRunner#reportHealthStatus bug fix
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5567:
---
Attachment: (was: YARN-5567.001.patch)
> NodeHealthScriptRunner#reportHealthStatus bug fix
> -
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5567) NodeHealthScriptRunner#reportHealthStatus bug fix
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5567:
---
Attachment: YARN-5567.001.patch
> NodeHealthScriptRunner#reportHealthStatus bug fix
> -
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5567.001.patch
>
>
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440067#comment-15440067 ] Ray Chiang commented on YARN-5549: -- +1 for removing it. Makes the review much cleaner. > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch, YARN-5549.004.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5567) NodeHealthScriptRunner#reportHealthStatus bug fix
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5567:
---
Attachment: (was: YARN-5567.001.patch)
> NodeHealthScriptRunner#reportHealthStatus bug fix
> -
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5567) NodeHealthScriptRunner#reportHealthStatus bug fix
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5567:
---
Attachment: YARN-5567.001.patch
> NodeHealthScriptRunner#reportHealthStatus bug fix
> -
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5567.001.patch
>
>
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Moved] (YARN-5567) NodeHealthScriptRunner#reportHealthStatus bug fix
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu moved MAPREDUCE-6770 to YARN-5567:
---
Component/s: (was: nodemanager)
nodemanager
Key: YARN-5567 (was: MAPREDUCE-6770)
Project: Hadoop YARN (was: Hadoop Map/Reduce)
> NodeHealthScriptRunner#reportHealthStatus bug fix
> -
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5221) Expose UpdateResourceRequest API to allow AM to request for change in container properties
[
https://issues.apache.org/jira/browse/YARN-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440046#comment-15440046
]
Subru Krishnan commented on YARN-5221:
--
I looked at the patch. +1 from my side (pending Yetus as the reports seem to
have expired), just a few minor comments:
* I think it is better to retain the old constructor of
{{ContainerTokenIdentifier}} and call the new one with implicit 0 version to
avoid changes to *MRApp* and possibly other applications. This will also avoid
changing existing tests that don't use container versioning.
* Nit: will be good to call out in the javadocs of {{Container}} that
versions are incremented as a side effect of update requests.
* Is it possible to use enum in {{UpdateContainerRequest}} to strictly
enforce only one aspect changes?
* Nit: You can use @link instead of in javadocs of
{{UpdateContainerRequest}} as there's is direct compile dependency.
A thought - can we defer splitting increase/decreaseRequests to the actual
scheduler as now we will have to propagate every new changeable container
attribute?
> Expose UpdateResourceRequest API to allow AM to request for change in
> container properties
> --
>
> Key: YARN-5221
> URL: https://issues.apache.org/jira/browse/YARN-5221
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5221.001.patch, YARN-5221.002.patch,
> YARN-5221.003.patch, YARN-5221.004.patch, YARN-5221.005.patch,
> YARN-5221.006.patch, YARN-5221.007.patch, YARN-5221.008.patch
>
>
> YARN-1197 introduced APIs to allow an AM to request for Increase and Decrease
> of Container Resources after initial allocation.
> YARN-5085 proposes to allow an AM to request for a change of Container
> ExecutionType.
> This JIRA proposes to unify both of the above into an Update Container API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[
https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440022#comment-15440022
]
Hadoop QA commented on YARN-5549:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 56s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 18s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m 33s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 57s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12825707/YARN-5549.003.patch |
| JIRA Issue | YARN-5549 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 4c1da263b003 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8b7adf4 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12907/testReport/ |
| modules | C:
[jira] [Updated] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5549: --- Attachment: YARN-5549.004.patch Good point. I'd much prefer not to add a new config param, especially for something as trivial as this. If the command information is needed for debugging, it's available from the job.xml file. > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch, YARN-5549.004.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5327) API changes required to support recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439956#comment-15439956 ] Subru Krishnan commented on YARN-5327: -- The latest patch LGTM. [~leftnoteasy], do you want to take a quick look to see if your concerns are addressed before I go ahead and commit it? > API changes required to support recurring reservations in the YARN > ReservationSystem > > > Key: YARN-5327 > URL: https://issues.apache.org/jira/browse/YARN-5327 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Attachments: YARN-5327.001.patch, YARN-5327.002.patch, > YARN-5327.003.patch, YARN-5327.004.patch, YARN-5327.005.patch, > YARN-5327.006.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to track the changes needed > in ApplicationClientProtocol to accomplish it. Please refer to the design doc > in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439951#comment-15439951 ] Vinod Kumar Vavilapalli commented on YARN-5549: --- bq. I don't have any objections to moving the command message to DEBUG. +1. Let's just move it to DEBUG or better just remove it - there is no need for a config option here. > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3671) Integrate Federation services with ResourceManager
[
https://issues.apache.org/jira/browse/YARN-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439920#comment-15439920
]
Hadoop QA commented on YARN-3671:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 55s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 41s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 8s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 50s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 45s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 3
new + 290 unchanged - 1 fixed = 293 total (was 291) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 33s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m 24s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 42s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12825702/YARN-3671-YARN-2915-v3.patch
|
| JIRA Issue | YARN-3671 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5b0280fb78b8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / 256034d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12905/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12905/testReport/ |
| modules | C:
[jira] [Commented] (YARN-679) add an entry point that can start any Yarn service
[
https://issues.apache.org/jira/browse/YARN-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439909#comment-15439909
]
Daniel Templeton commented on YARN-679:
---
I made it a bit farther. Additional comments:
* I see that the pattern of the code from the cause trumping the code parameter
shows up in several other places. I see what's you're going for, but I really
dislike the idea of passing in two arguments for the same data with one being
ignored. I'm open to discuss it.
* Should the {{ServiceStateException}} and {{ExitUtil.ExitException}} have a
constructor that only takes a message and exit code?
* Shouldn't need the period after {code}* {@inheritDoc}.{code}
* I'd rather see {code} private static final Logger LOG =
LoggerFactory.getLogger(
HadoopUncaughtExceptionHandler.class);{code} break on the equals instead
of the paren.
Here are also some language issues from the package docs:
* Should have an "and" instead of a comma: {code} down the CLI arguments, to
execute an operation without having to{code}
* Should be "out" not "our": {code}exit if it times our or a second interrupt
is received.{code}
* I'd rephrase: {code}It will instantiate the service classname provided, by
zero-args constructor. Or, if none is available, falling back to a constructor
that takes a {@code String} as its parameter, on the assumption that the
parameter is the service name.{code} as {code}It will instantiate the service
classname provided, using the no-args constructor, or if no such constructor is
available, it will fall back to a constructor with a single {@code String}
parameter, passing the service name as the parameter value.{code}
* Missing a close paren before the comma here: {code}{@code STATE.STOPPED},
escalated into a non-zero return code{code}
* Extra space before the dash (and add another hyphen to make it a dash or use
{{}}): {code}(prepare configuration files -covered later){code}.
* Same here: {code}and control-C signals -calling {@code stop()} on the service
when signalled{code}
* And here: {code}parse the command line -it just becomes the responsibility of
the{code}
* And here: {code}(prepare configuration files --covered later){code}
* Here the hyphen should be a comma: {code}and start it -triggering shutdown
when signalled{code}
* Same here: {code}It adds two methods to the service interface -and hence two
new features{code}, though I suppose a dash could make sense...
* And here: {code}returned by the method -so services may signal failures
simply by returning{code}
* And here: {code}Exceptions are converted into exit codes -but rather than
simply{code}
* "Signalled" is misspelled several times. Should be "signaled"
* {code}(there is way covered later){code} is missing an "a"
* Here: {code}commands can be implemented as such services, so integrating with
YARN's{code}, the "so" should be a "thus"
* Same here: {code}initialized, so allowing services to tune their
configuration data before{code}
* This hyphen is spurious: {code}-which may be needed to trigger the injection
of HDFS or YARN resources{code}
* Same here: {code}of returning error codes to signal failures -and for {code}
* And here: {code}these classes -simply to force in the common resources{code}
* And here: {code}files are merged -with the latest file on the command line
being the{code}
* Remove the space before the comma: {code}have a "something went wrong" exit
code , exceptions may{code}
* "such" should be "the": {code}the service instance and such like{code}
* Missing an "i" in "is": {code}exit code of 0 s created{code}
* Here the "-so" should just be a comma: {code}stop the service if a shutdown
request is received -so ensuring that{code}
Maybe after the weekend I'll be up to finishing the review. :)
> add an entry point that can start any Yarn service
> --
>
> Key: YARN-679
> URL: https://issues.apache.org/jira/browse/YARN-679
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: YARN-679-001.patch, YARN-679-002.patch,
> YARN-679-002.patch, YARN-679-003.patch, YARN-679-004.patch,
> YARN-679-005.patch, YARN-679-006.patch, YARN-679-007.patch,
> YARN-679-008.patch, YARN-679-009.patch, YARN-679-010.patch,
> YARN-679-011.patch, org.apache.hadoop.servic...mon 3.0.0-SNAPSHOT API).pdf
>
> Time Spent: 72h
> Remaining Estimate: 0h
>
> There's no need to write separate .main classes for every Yarn service, given
> that the startup mechanism should be identical: create, init, start, wait for
> stopped -with an interrupt handler to trigger a clean shutdown on a control-c
> interrupt.
> Provide one that takes any classname, and a list of config files/options
--
This
[jira] [Commented] (YARN-5560) Clean up bad exception catching practices in TestYarnClient
[
https://issues.apache.org/jira/browse/YARN-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439861#comment-15439861
]
Hadoop QA commented on YARN-5560:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
The patch generated 0 new + 63 unchanged - 2 fixed = 63 total (was 65) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 19s
{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 33s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12825706/YARN-5560.v3.patch |
| JIRA Issue | YARN-5560 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 56c2e05d943e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / cde3a00 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12906/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12906/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Clean up bad exception catching practices in TestYarnClient
> ---
>
> Key: YARN-5560
> URL: https://issues.apache.org/jira/browse/YARN-5560
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Sean Po
>Assignee: Sean Po
> Attachments: YARN-5560.v1.patch, YARN-5560.v2.patch,
> YARN-5560.v3.patch
>
>
> In TestYarnClient, tests commonly wrap methods that throw
[jira] [Commented] (YARN-5506) Inconsistent logging content and logging level for distributed shell
[
https://issues.apache.org/jira/browse/YARN-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439793#comment-15439793
]
Nemo Chen commented on YARN-5506:
-
I agree. Thanks for the replies.
> Inconsistent logging content and logging level for distributed shell
>
>
> Key: YARN-5506
> URL: https://issues.apache.org/jira/browse/YARN-5506
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Affects Versions: 2.7.2
>Reporter: Nemo Chen
>Assignee: Vrushali C
> Labels: easyfix, easytest
>
> Similar to MAPREDUCE-6752, in file:
> hadoop-rel-release-2.7.2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
> Line 313, the logging purpose is for debugging, but the level is set to info.
> {code:borderStyle=solid}
> /**
>* Dump out contents of $CWD and the environment to stdout for debugging
>*/
> private void dumpOutDebugInfo() {
> LOG.info("Dump debug output");
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5506) Inconsistent logging content and logging level for distributed shell
[
https://issues.apache.org/jira/browse/YARN-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439792#comment-15439792
]
Nemo Chen commented on YARN-5506:
-
I agree. Thanks for the replies!
> Inconsistent logging content and logging level for distributed shell
>
>
> Key: YARN-5506
> URL: https://issues.apache.org/jira/browse/YARN-5506
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Affects Versions: 2.7.2
>Reporter: Nemo Chen
>Assignee: Vrushali C
> Labels: easyfix, easytest
>
> Similar to MAPREDUCE-6752, in file:
> hadoop-rel-release-2.7.2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
> Line 313, the logging purpose is for debugging, but the level is set to info.
> {code:borderStyle=solid}
> /**
>* Dump out contents of $CWD and the environment to stdout for debugging
>*/
> private void dumpOutDebugInfo() {
> LOG.info("Dump debug output");
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5549: --- Attachment: YARN-5549.003.patch I just adjusted the wording to make it sound less like the new property can be applied on a per-job basis. > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5560) Clean up bad exception catching practices in TestYarnClient
[
https://issues.apache.org/jira/browse/YARN-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Po updated YARN-5560:
--
Attachment: YARN-5560.v3.patch
You're welcome [~jlowe], thanks for reviewing! V3 of the patch addresses your
previous comment.
> Clean up bad exception catching practices in TestYarnClient
> ---
>
> Key: YARN-5560
> URL: https://issues.apache.org/jira/browse/YARN-5560
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Sean Po
>Assignee: Sean Po
> Attachments: YARN-5560.v1.patch, YARN-5560.v2.patch,
> YARN-5560.v3.patch
>
>
> In TestYarnClient, tests commonly wrap methods that throw exceptions in a try
> catch statement similar to the following:
> {code}
> try {
> client.submitApplication(context);
> } catch (Exception e) {
> Assert.fail("Exception is not expected.");
> }
> {code}
> This hides useful error messages, and surfaces less helpful ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5505) Create an agent-less docker provider in the native-services framework
[ https://issues.apache.org/jira/browse/YARN-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-5505: -- Summary: Create an agent-less docker provider in the native-services framework (was: Create an agent-less docker provider) > Create an agent-less docker provider in the native-services framework > - > > Key: YARN-5505 > URL: https://issues.apache.org/jira/browse/YARN-5505 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-5505-yarn-native-services.001.patch > > > The Slider AM has a pluggable portion called a provider. Currently the only > provider implementation is the agent provider which contains the bulk of the > agent-related Java code. We can implement a docker provider that does not use > the agent and gets information it needs directly from the NM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3671) Integrate Federation services with ResourceManager
[ https://issues.apache.org/jira/browse/YARN-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-3671: - Attachment: YARN-3671-YARN-2915-v3.patch [~jianhe], I looked into it & seems like we should be able to reuse RM_CLUSTER_ID safely as you suggested. So I have replaced FEDERATION_SUBCLUSTER_ID with RM_CLUSTER_ID in v3 of the patch. > Integrate Federation services with ResourceManager > -- > > Key: YARN-3671 > URL: https://issues.apache.org/jira/browse/YARN-3671 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3671-YARN-2915-v1.patch, > YARN-3671-YARN-2915-v2.patch, YARN-3671-YARN-2915-v3.patch > > > This JIRA proposes adding the ability to turn on Federation services like > StateStore, cluster membership heartbeat etc in the RM -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439696#comment-15439696 ] Ray Chiang commented on YARN-5549: -- I don't have any objections to moving the command message to DEBUG. Anyone else? > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5560) Clean up bad exception catching practices in TestYarnClient
[
https://issues.apache.org/jira/browse/YARN-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439630#comment-15439630
]
Jason Lowe commented on YARN-5560:
--
Thanks, Sean! Patch looks good overall with just one minor nit. Since we're
cleaning up bad exception catching practices, this one can be cleaned up as
well.
{code}
private void waitTillAccepted(YarnClient rmClient, ApplicationId appId,
boolean unmanagedApplication)
throws Exception {
try {
[...]
} catch (Exception ex) {
throw new Exception(ex);
}
{code}
waitTillAccepted already can throw Exception, so catching an Exception to wrap
it in an Exception is redundant.
> Clean up bad exception catching practices in TestYarnClient
> ---
>
> Key: YARN-5560
> URL: https://issues.apache.org/jira/browse/YARN-5560
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Sean Po
>Assignee: Sean Po
> Attachments: YARN-5560.v1.patch, YARN-5560.v2.patch
>
>
> In TestYarnClient, tests commonly wrap methods that throw exceptions in a try
> catch statement similar to the following:
> {code}
> try {
> client.submitApplication(context);
> } catch (Exception e) {
> Assert.fail("Exception is not expected.");
> }
> {code}
> This hides useful error messages, and surfaces less helpful ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2774) shared cache service should authorize calls properly
[ https://issues.apache.org/jira/browse/YARN-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439542#comment-15439542 ] Chris Trezzo commented on YARN-2774: bq. EDIT: Never mind, looks like those were tickets for services that need authentication. Yes correct. All of the services are now in place, but authentication needs to be implemented for them. > shared cache service should authorize calls properly > > > Key: YARN-2774 > URL: https://issues.apache.org/jira/browse/YARN-2774 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sangjin Lee > > The shared cache manager (SCM) services should authorize calls properly. > Currently, the uploader service (done in YARN-2186) does not authorize calls > to notify the SCM on newly uploaded resource. Proper security/authorization > needs to be done in this RPC call. Also, the use/release calls (done in > YARN-2188) and the scmAdmin commands (done in YARN-2189) are not properly > authorized. The SCM UI done in YARN-2203 as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5540) scheduler spends too much time looking at empty priorities
[
https://issues.apache.org/jira/browse/YARN-5540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439516#comment-15439516
]
Hadoop QA commented on YARN-5540:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 6 unchanged - 2 fixed = 6 total (was 8) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 37s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 34s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12825677/YARN-5540.002.patch |
| JIRA Issue | YARN-5540 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d97abe591d5c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9ef632f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12904/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12904/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> scheduler spends too much time looking at empty priorities
> --
>
> Key: YARN-5540
> URL: https://issues.apache.org/jira/browse/YARN-5540
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, fairscheduler, resourcemanager
>
[jira] [Commented] (YARN-5373) NPE listing wildcard directory in containerLaunch
[
https://issues.apache.org/jira/browse/YARN-5373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439513#comment-15439513
]
Hudson commented on YARN-5373:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10356 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10356/])
YARN-5373. NPE listing wildcard directory in containerLaunch. (Daniel (kasha:
rev cde3a00526c562a500308232e2b93498d22c90d7)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
> NPE listing wildcard directory in containerLaunch
> -
>
> Key: YARN-5373
> URL: https://issues.apache.org/jira/browse/YARN-5373
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.9.0
>Reporter: Haibo Chen
>Assignee: Daniel Templeton
>Priority: Blocker
> Fix For: 2.9.0
>
> Attachments: YARN-5373.001.patch, YARN-5373.002.patch,
> YARN-5373.003.patch
>
>
> YARN-4958 added support for wildcards in file localization. It introduces a
> NPE
> at
> {code:java}
> for (File wildLink : directory.listFiles()) {
> sb.symlink(new Path(wildLink.toString()), new Path(wildLink.getName()));
> }
> {code}
> When directory.listFiles returns null (only happens in a secure cluster), NPE
> will cause the container fail to launch.
> Hive, Oozie jobs fail as a result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5373) NPE listing wildcard directory in containerLaunch
[
https://issues.apache.org/jira/browse/YARN-5373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439476#comment-15439476
]
Karthik Kambatla commented on YARN-5373:
The latest patch looks good. +1. Checking this in.
> NPE listing wildcard directory in containerLaunch
> -
>
> Key: YARN-5373
> URL: https://issues.apache.org/jira/browse/YARN-5373
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.9.0
>Reporter: Haibo Chen
>Assignee: Daniel Templeton
>Priority: Blocker
> Attachments: YARN-5373.001.patch, YARN-5373.002.patch,
> YARN-5373.003.patch
>
>
> YARN-4958 added support for wildcards in file localization. It introduces a
> NPE
> at
> {code:java}
> for (File wildLink : directory.listFiles()) {
> sb.symlink(new Path(wildLink.toString()), new Path(wildLink.getName()));
> }
> {code}
> When directory.listFiles returns null (only happens in a secure cluster), NPE
> will cause the container fail to launch.
> Hive, Oozie jobs fail as a result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5566) client-side NM graceful decom doesn't trigger when jobs finish
[ https://issues.apache.org/jira/browse/YARN-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439437#comment-15439437 ] Robert Kanter commented on YARN-5566: - I've been super busy and haven't had a chance to look at YARN-4676 again. I've actually been testing the client-side graceful decom without YARN-4676 applied, so the logic that was removed by it is actually still there. I agree that UNHEALTHY isn't a state that should occur if the initial state is DECOMMISSIONING; as you pointed out, the code doesn't go to UNHEALTHY from DECOMMISSIONG. That's partly why I was unsure why my patch fixed the problem; the node never goes to UNHEALTHY, so it doesn't make sense why that would help. I'll try to dig into this some more. > client-side NM graceful decom doesn't trigger when jobs finish > -- > > Key: YARN-5566 > URL: https://issues.apache.org/jira/browse/YARN-5566 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: YARN-5566.001.patch > > > I was testing the client-side NM graceful decommission and noticed that it > was always waiting for the timeout, even if all jobs running on that node (or > even the cluster) had already finished. > For example: > # JobA is running with at least one container on NodeA > # User runs client-side decom on NodeA at 5:00am with a timeout of 3 hours > --> NodeA enters DECOMMISSIONING state > # JobA finishes at 6:00am and there are no other jobs running on NodeA > # User's client reaches the timeout at 8:00am, and forcibly decommissions > NodeA > NodeA should have decommissioned at 6:00am. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5538) Apply SLIDER-875 to yarn-native-services for supporting a composite app-of-apps
[ https://issues.apache.org/jira/browse/YARN-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-5538: -- Summary: Apply SLIDER-875 to yarn-native-services for supporting a composite app-of-apps(was: Apply SLIDER-875 to yarn-native-services) > Apply SLIDER-875 to yarn-native-services for supporting a composite > app-of-apps > - > > Key: YARN-5538 > URL: https://issues.apache.org/jira/browse/YARN-5538 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: yarn-native-services > > Attachments: YARN-5538-yarn-native-services.001.patch > > > The SLIDER-875 feature branch was recently merged into Slider develop and it > would be good to have these changes in the yarn-native-services branch as > well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
[ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439387#comment-15439387 ] Sunil G commented on YARN-4945: --- Thanks [~eepayne] for the comments about the interaction b/w inter-queue and intra-queue together. Its a really a case to consider. While working on POC, [~leftnoteasy] has mentioned about a case of normalization. We have to consider that inter-queue preemption is done while doing preemption calculation and selection for intra-queue. In the initial version of POC, i tried to achieve the basic framework and a working prototype. But for the version, we need to normalize pre-selected containers before each round in intra-queue preemption. (i think we can consider user-limit and priority only for now) On the same note, we also need to consider case where selected containers are not rejected in subsequent rounds. Also if same containers are again re-selected in next rounds, we can consider those containers are already selected and deduct from resourceToObtain. These cases are handled as of today. I think I can add the impl details of this scenario also in the document so that it can come in first cut. Thoughts? > [Umbrella] Capacity Scheduler Preemption Within a queue > --- > > Key: YARN-4945 > URL: https://issues.apache.org/jira/browse/YARN-4945 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan > Attachments: Intra-Queue Preemption Use Cases.pdf, > IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.patch > > > This is umbrella ticket to track efforts of preemption within a queue to > support features like: > YARN-2009. YARN-2113. YARN-4781. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5540) scheduler spends too much time looking at empty priorities
[
https://issues.apache.org/jira/browse/YARN-5540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated YARN-5540:
-
Attachment: YARN-5540.002.patch
Updating the patch to use a ConcurrentSkipListMap instead of a TreeMap. This
is not going to be as cheap as having the iterator do the removal, but it's far
less code change and more robust if we allow other threads to update the
requests as the scheduler is examining them.
I also updated the TODO comment and added a check for the CME possibility which
catches the error from the previous patch.
> scheduler spends too much time looking at empty priorities
> --
>
> Key: YARN-5540
> URL: https://issues.apache.org/jira/browse/YARN-5540
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, fairscheduler, resourcemanager
>Affects Versions: 2.7.2
>Reporter: Nathan Roberts
>Assignee: Jason Lowe
> Attachments: YARN-5540.001.patch, YARN-5540.002.patch
>
>
> We're starting to see the capacity scheduler run out of scheduling horsepower
> when running 500-1000 applications on clusters with 4K nodes or so.
> This seems to be amplified by TEZ applications. TEZ applications have many
> more priorities (sometimes in the hundreds) than typical MR applications and
> therefore the loop in the scheduler which examines every priority within
> every running application, starts to be a hotspot. The priorities appear to
> stay around forever, even when there is no remaining resource request at that
> priority causing us to spend a lot of time looking at nothing.
> jstack snippet:
> {noformat}
> "ResourceManager Event Processor" #28 prio=5 os_prio=0 tid=0x7fc2d453e800
> nid=0x22f3 runnable [0x7fc2a8be2000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.getResourceRequest(SchedulerApplicationAttempt.java:210)
> - eliminated <0x0005e73e5dc0> (a
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:852)
> - locked <0x0005e73e5dc0> (a
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp)
> - locked <0x0003006fcf60> (a
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:527)
> - locked <0x0003001b22f8> (a
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:415)
> - locked <0x0003001b22f8> (a
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1224)
> - locked <0x000300041e40> (a
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-679) add an entry point that can start any Yarn service
[
https://issues.apache.org/jira/browse/YARN-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439375#comment-15439375
]
Junping Du commented on YARN-679:
-
Agree with [~templedf] that this is well written javadoc that can be good
example for reference in future. :)
Good to see we finally figure out Jenkins' problem last week. Sorry I haven't
figure out too much time this week to review it thoroughly. Just have some
level comments here:
1. It looks like all changes are in hadoop-common project currently. If so,
shall we turn it into Hadoop JIRA instead of a YARN issue? Or we think these
APIs only get used for YARN application. If so, may be we should consider to
refactor most code to under hadoop-yarn-project (like: yarn-api)?
2. The lifecycle of yarn service looks good to me. However, I haven't see an
detail example to show case how an application can use these code to launch
yarn service. Do we have a quick dummy application here or we can plan to add
one in future.
3. I saw we were wrapping a lot of error code here around all kinds of
exceptions or HTTP code.
{noformat}
+ *
+ *0-10: general command issues
+ * 30-39: equivalent to the 3XX responses, where those responses are
+ * considered errors by the application.
+ * 40-49: client-side/CLI/config problems
+ * 50-59: service-side problems.
+ * 60+ : application specific error codes
+ *
{noformat}
It sounds like 11-20 is vacant. Is that reserved for other purpose. Also, if
error code is used out (like service side problem is more than 10 kinds). How
can we extend according to current design?
I need more time for check more details of the patch which is very huge now.
> add an entry point that can start any Yarn service
> --
>
> Key: YARN-679
> URL: https://issues.apache.org/jira/browse/YARN-679
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: YARN-679-001.patch, YARN-679-002.patch,
> YARN-679-002.patch, YARN-679-003.patch, YARN-679-004.patch,
> YARN-679-005.patch, YARN-679-006.patch, YARN-679-007.patch,
> YARN-679-008.patch, YARN-679-009.patch, YARN-679-010.patch,
> YARN-679-011.patch, org.apache.hadoop.servic...mon 3.0.0-SNAPSHOT API).pdf
>
> Time Spent: 72h
> Remaining Estimate: 0h
>
> There's no need to write separate .main classes for every Yarn service, given
> that the startup mechanism should be identical: create, init, start, wait for
> stopped -with an interrupt handler to trigger a clean shutdown on a control-c
> interrupt.
> Provide one that takes any classname, and a list of config files/options
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5503) [YARN-3368] Add missing hidden files in webapp folder
[ https://issues.apache.org/jira/browse/YARN-5503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439349#comment-15439349 ] Sunil G commented on YARN-5503: --- As discussed, ASF warnings are not related to the patch. Hence its fine. Also local test with rat plugin too came fine. Still as discussed with Varun, could you please split patch so that .gitignore changes can be done separately as it touching all unit test cases from base folder and those are failing. Thoughts? > [YARN-3368] Add missing hidden files in webapp folder > - > > Key: YARN-5503 > URL: https://issues.apache.org/jira/browse/YARN-5503 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: YARN-5503-YARN-3368-0001.patch, > YARN-5503-YARN-3368-0002.patch, YARN-5503-YARN-3368-0003.patch, > YARN-5503-YARN-3368-0004.patch, YARN-5503-YARN-3368.0005.patch > > > - Feel it might be good to have a readme file with the basic instructions. > - Just noticed that the hidden files that must be present in the base > directory of ember app, are missing. Most of them are used for configuration, > and when missing the default vakues would be used by ember. > -- They include - .bowerrc, .editorconfig, .ember-cli, .gitignore, .jshintrc, > .travis.yml, .watchmanconfig -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5557) Add localize API to the ContainerManagementProtocol
[ https://issues.apache.org/jira/browse/YARN-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439341#comment-15439341 ] Hudson commented on YARN-5557: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10355 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10355/]) YARN-5557. Add localize API to the ContainerManagementProtocol. (junping_du: rev 9ef632f3b0b0e0808116cd1c7482a205b7ebef7d) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/ResourceLocalizationRequest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/ResourceLocalizationRequestPBImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/ResourceLocalizationResponsePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ContainerManagementProtocolPBClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLaunchRPC.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ContainerManagementProtocol.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/ResourceLocalizationResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerResourceIncreaseRPC.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/containermanagement_protocol.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ContainerManagementProtocolPBServiceImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java > Add localize API to the ContainerManagementProtocol > --- > > Key: YARN-5557 > URL: https://issues.apache.org/jira/browse/YARN-5557 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5557.1.patch, YARN-5557.2.patch, YARN-5557.3.patch, > YARN-5557.4.patch > > > A new localize API for localizing new resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5566) client-side NM graceful decom doesn't trigger when jobs finish
[
https://issues.apache.org/jira/browse/YARN-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439340#comment-15439340
]
Junping Du commented on YARN-5566:
--
Hi [~rkanter], thanks for reporting this issue. Unhealthy state is not a
expected one in StatusUpdateWhenHealthyTransition if node's initial state is
decommissioning. I am not exactly sure how update changes like attached patch
could help here but there is indeed something wrong with our latest change.
First, I think this is a regression issue caused by YARN-4676
(https://github.com/apache/hadoop/commit/0da69c324dee9baab0f0b9700db1cc5b623f8421#diff-29befd5a1922a2121b26766561f9e447).
You can see we were removing the logic of detecting of app finish in
StatusUpdateWhenHealthyTransition. We should add it back (in branch-2.9).
I assume you do your test in bit of branch-2. Isn't it?
Another issue we should fix here (in either 2.8 or 2.9) is: if NM keep
reporting unhealthy after it is in decommissioning stage, then it will only
ends up as decommissioning without get chance to move to decommissioned.
{noformat}
if (!remoteNodeHealthStatus.getIsNodeHealthy()) {
LOG.info("Node " + rmNode.nodeId +
" reported UNHEALTHY with details: " +
remoteNodeHealthStatus.getHealthReport());
// if a node in decommissioning receives an unhealthy report,
// it will keep decommissioning.
if (isNodeDecommissioning) {
return NodeState.DECOMMISSIONING;
} else {
{noformat}
Instead of return NodeState.DECOMMISSIONING directly, we should also check
running apps on that node and return decommissioned if no running apps. Isn't?
> client-side NM graceful decom doesn't trigger when jobs finish
> --
>
> Key: YARN-5566
> URL: https://issues.apache.org/jira/browse/YARN-5566
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-5566.001.patch
>
>
> I was testing the client-side NM graceful decommission and noticed that it
> was always waiting for the timeout, even if all jobs running on that node (or
> even the cluster) had already finished.
> For example:
> # JobA is running with at least one container on NodeA
> # User runs client-side decom on NodeA at 5:00am with a timeout of 3 hours
> --> NodeA enters DECOMMISSIONING state
> # JobA finishes at 6:00am and there are no other jobs running on NodeA
> # User's client reaches the timeout at 8:00am, and forcibly decommissions
> NodeA
> NodeA should have decommissioned at 6:00am.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5557) Add localize API to the ContainerManagementProtocol
[ https://issues.apache.org/jira/browse/YARN-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439284#comment-15439284 ] Arun Suresh commented on YARN-5557: --- agreed.. thanks again for the patch [~jianhe].. > Add localize API to the ContainerManagementProtocol > --- > > Key: YARN-5557 > URL: https://issues.apache.org/jira/browse/YARN-5557 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5557.1.patch, YARN-5557.2.patch, YARN-5557.3.patch, > YARN-5557.4.patch > > > A new localize API for localizing new resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5551) Ignore deleted file mapping from memory computation when smaps is enabled
[
https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439274#comment-15439274
]
Jason Lowe commented on YARN-5551:
--
bq. Memory pressure could cause either of them to spill to disk, so it doesn't
make sense to me why we would count them differently.
Hmm. I guess they will behave differently in practice because a file-backed
region of memory could pause the process if it dirties too many pages and
triggers an I/O flush, whereas that's not going to happen if the memory is
simply backed by the system's swap device.
So I suppose ignoring the private dirty field and only focusing on anonymous
pages even with plenty of swap makes sense because we're equating private dirty
with the buffer cache and we don't charge other processes for their use of the
buffer cache.
> Ignore deleted file mapping from memory computation when smaps is enabled
> -
>
> Key: YARN-5551
> URL: https://issues.apache.org/jira/browse/YARN-5551
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: YARN-5551.branch-2.001.patch
>
>
> Currently deleted file mappings are also included in the memory computation
> when SMAP is enabled. For e.g
> {noformat}
> 7f612004a000-7f612004c000 rw-s 00:10 4201507513
> /dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-521969216_162_734673185
> (deleted)
> Size: 8 kB
> Rss: 4 kB
> Pss: 2 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 4 kB
> Private_Clean: 0 kB
> Private_Dirty: 0 kB
> Referenced:4 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize:4 kB
> MMUPageSize: 4 kB
> 7fbf2800-7fbf6800 rw-s 08:02 11927571
> /tmp/7298569189125604642/arena-1291157252088664681.cache (deleted)
> Size:1048576 kB
> Rss: 17288 kB
> Pss: 17288 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 232 kB
> Private_Dirty: 17056 kB
> Referenced:17288 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize:4 kB
> MMUPageSize: 4 kB
> {noformat}
> It would be good to exclude these from getSmapBasedRssMemorySize()
> computation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5557) Add localize API to the ContainerManagementProtocol
[ https://issues.apache.org/jira/browse/YARN-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439275#comment-15439275 ] Jian He commented on YARN-5557: --- I'd like to do it in trunk as I only need one more patch to get the core functionality ready, and the overall size is not big. > Add localize API to the ContainerManagementProtocol > --- > > Key: YARN-5557 > URL: https://issues.apache.org/jira/browse/YARN-5557 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5557.1.patch, YARN-5557.2.patch, YARN-5557.3.patch, > YARN-5557.4.patch > > > A new localize API for localizing new resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439264#comment-15439264
]
Sunil G commented on YARN-4945:
---
Thanks [~eepayne] for the use cases.
Overall I feel the use cases looks fine for me,. I have looked the doc as well.
Few comments from my end
1.
{noformat}
When one (or more) user(s) are below their minimun-user-limit-percent and one
(or
more) user(s) are above their minimum-user-limit-percent, resources will be
preempted after a configurable time period from the user(s) which are above
their minimumuser-limit-percent.
{noformat}
>>I think we might need to come with a limit on how much resource can be
>>preempted from over-utilizing users's apps. WE do have
>>max-preemption-per-round. But sometimes it may be more as it may be
>>configured for inter-queue. Since we are sharing this config, i think we can
>>have a config to limit the preemption for user-limit. For priority, i have
>>considered a certain limit to control this scenario. Thoughts?
2.
{noformat}
App0 in QUEUE0 is consuming 50 extra resources and is not releasing them. But,
since QUEUE0 is not preemptable, inter-queue preemption will not free the 50
resources from QUEUE0. In QUEUE1, User2 is guaranteed 50 resources, but
User1 is also, and User1 is not going over its minimum-user-limitpercent,
so User1 is not violating any policy. Therefore, the
user-limit-percentintra-queue-preemption
policy does not preempt any resources from User1.
{noformat}
>> Overall this point make sense. This is strictly intra-queue model and we
>> need not have to consider across queue demand and its shortages.
3.
{noformat}
1.3.1.3.1. TBD: I could see this going the other way. That is, we may want the
policy to preempt from App1 until both users have an equal number of
resources.
{noformat}
>> I mostly agree with you here. Once app1's containers are released, app2 can
>> get resources. CS will ensure same, I dont think preemption module need to
>> do that.
4.
bq.Should there be some (possibly configurable) limit on the percent of
containers preempted from App1?
Instead of limiting from each app, I consider only a certain % of high priority
app's demand. This can help to avoid over preemption cases. If we limit
preempting container from an app, a higher priority app may loose some
container. So for now, its better to clear lowest priority apps completely
except its AM. On same line, apps could save some of its containers (critical,
long running). For this, app itself can mark such containers. This will come as
part of first class services, and we could improve preemption after same.
5.
bq.the priority preemption policy will not preempt containers from the lower
priority app if it would cause the lower priority app to go below the user’s
minimum-user-limit-percent guarantee.
I will not consider preemption demand from a high priority if that app is
already crossing the user-limit. Such apps will be skipped while calculating
demand. I have added this support in POC patch itself.
Overall these cases make sense, and thank you very much or sharing same. Its
very good to validate and consider all problem statements at this stage itself.
:)
I have not looked your latest comment, i will help to comment on same in a
shortwhile.
> [Umbrella] Capacity Scheduler Preemption Within a queue
> ---
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf,
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5557) Add localize API to the ContainerManagementProtocol
[ https://issues.apache.org/jira/browse/YARN-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439248#comment-15439248 ] Junping Du commented on YARN-5557: -- bq. but I am wondering if it would be better to have this checked into a branch rather than trunk ? I think this patch should be OK to land on either trunk or a feature branch. Normally, we would suggest to develop a feature in a feature branch in following cases: 1. this feature is a huge effort and very risky to conflict with other ongoing efforts. 2. some main contributors of that feature is not committer yet, making them branch committer could accelerate feature development. IMO, this feature doesn't belongs to two cases here. Anyway, I am flexible with both approaches here. But I would prefer to respect feature owner [~jianhe]'s choices given he should have better understanding on this feature and take the following pain after making decisions. [~jianhe], what's your choice here? > Add localize API to the ContainerManagementProtocol > --- > > Key: YARN-5557 > URL: https://issues.apache.org/jira/browse/YARN-5557 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5557.1.patch, YARN-5557.2.patch, YARN-5557.3.patch, > YARN-5557.4.patch > > > A new localize API for localizing new resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5551) Ignore deleted file mapping from memory computation when smaps is enabled
[
https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439246#comment-15439246
]
Jason Lowe commented on YARN-5551:
--
bq. Yes, deleted files is a red-herring
OK, cool. That's been my main issue with the JIRA's proposed change.
bq. If I understand page-writeback.c correctly, the blocking operation would be
the page fault on a memory block which is missing in memory.
Good, so it looks like users can't abuse an mmap'd region any more than they
can abuse the buffer cache. Thanks for looking into this.
bq. If YARN counted only the "anonymous" pages as the "will be free'd a kill"
memory, it would give me a better way.
I'm torn on this proposal. My initial reaction to ignoring private dirty pages
is that a user can hide a lot of memory by creating their own, personal swap
file. For example, a user allocates 10G normally and touches it all. I think
we all agree that should count as 10G of usage. If the user changes the app to
back that memory with a private file on disk then it wouldn't count for
anything. That seems wrong since the user is charged differently based on
whether the memory is backed by public swap vs. private swap. Memory pressure
could cause either of them to spill to disk, so it doesn't make sense to me why
we would count them differently.
On the other hand, it would be a different situation if there was no swap
configured on the system. Memory pressure could push out the private dirty
pages but dirty anonymous pages are pinned until the process exits. Then I
think it makes sense to count them differently because they won't behave the
same under memory pressure.
> Ignore deleted file mapping from memory computation when smaps is enabled
> -
>
> Key: YARN-5551
> URL: https://issues.apache.org/jira/browse/YARN-5551
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: YARN-5551.branch-2.001.patch
>
>
> Currently deleted file mappings are also included in the memory computation
> when SMAP is enabled. For e.g
> {noformat}
> 7f612004a000-7f612004c000 rw-s 00:10 4201507513
> /dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-521969216_162_734673185
> (deleted)
> Size: 8 kB
> Rss: 4 kB
> Pss: 2 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 4 kB
> Private_Clean: 0 kB
> Private_Dirty: 0 kB
> Referenced:4 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize:4 kB
> MMUPageSize: 4 kB
> 7fbf2800-7fbf6800 rw-s 08:02 11927571
> /tmp/7298569189125604642/arena-1291157252088664681.cache (deleted)
> Size:1048576 kB
> Rss: 17288 kB
> Pss: 17288 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 232 kB
> Private_Dirty: 17056 kB
> Referenced:17288 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize:4 kB
> MMUPageSize: 4 kB
> {noformat}
> It would be good to exclude these from getSmapBasedRssMemorySize()
> computation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5557) Add localize API to the ContainerManagementProtocol
[ https://issues.apache.org/jira/browse/YARN-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439175#comment-15439175 ] Arun Suresh commented on YARN-5557: --- bq. I think all new added APIs are marked as Unstable. Which method you are talking about here? Aah... I think I was looking at one of the Impl / Test classes.. my bad.. +1 to the patch.. but I am wondering if it would be better to have this checked into a branch rather than trunk ? > Add localize API to the ContainerManagementProtocol > --- > > Key: YARN-5557 > URL: https://issues.apache.org/jira/browse/YARN-5557 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5557.1.patch, YARN-5557.2.patch, YARN-5557.3.patch, > YARN-5557.4.patch > > > A new localize API for localizing new resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439148#comment-15439148
]
Eric Payne commented on YARN-4945:
--
[~sunilg] and [~leftnoteasy], While I was writing the use cases, I thought
about the interaction between the inter-queue and intra-queue preemption
policies, and I want to get your thoughts. The following use case is not in the
doc, because I'm not sure how to handle it.
Consider this use case:
The cluster has 2 queues, both preemptable, both have configured max capacity
that can use the whole cluster.
||Queue Name||Queue Guaranteed Resources||Queue Max Resources||Queue
{{minimum-user-limit-percent}}||Queue Preemptable||
|root|200|200|N/A|N/A|
|{{QUEUE1}}|100|200|100%|Yes|
|{{QUEUE2}}|100|200|50%|Yes|
---
||Queue Name||User Name||App Name||Resources Used||Resources Guaranteed Per
User by
{{minimum-user-limit-percent}}||Pending Resources||
|{{QUEUE1}}|{{User1}}|{{App1}}|120|100|0|
|{{QUEUE2}}|{{User2}}|{{App2}}|80|50|0|
|{{QUEUE2}}|{{User3}}|{{App3}}|0|50|20|
# The inter-queue preemption policy sees that {{QUEUE2}} is underserved and is
asking for 20 resources, and {{QUEUE1}} is over-served by 20 resources, so it
preempts 20 resources from {{App1}}.
# The intra-queue preemption policy sees that {{User3}} is under its
{{minimum-user-limit-percent}} and is aksing for 20 resources, and {{User2}} is
over its {{minimum-user-limit-percent}}, so the intra-queue-preemption policy
preempts 20 resources from {{App2}}.
# The result of this scenario is that 20 resources are preempted when they
should not be.
The scenario I have laid out above assumes that intra-queue preemption did not
know about the 20 containers that are already preempted to fulfill the needs of
{{App3}} in {{QUEUE2}}. I think that the design doc tries to address this, and
assumes that the intra-queue preemption policy will be able to handle this use
case and will not preempt more containers when it is not necessary.
However, I am not so sure about that. In a more complicated scenario with
multiple over-served and multiple under-served queues, how will the intra-queue
preemption policy know that the containers that are already in the
{{selectedContainers}} list will be used to fulfill the needs of any specific
queue?
Please provide your thoughts.
> [Umbrella] Capacity Scheduler Preemption Within a queue
> ---
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf,
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5557) Add localize API to the ContainerManagementProtocol
[ https://issues.apache.org/jira/browse/YARN-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439140#comment-15439140 ] Junping Du commented on YARN-5557: -- [~asuresh], any further comments? > Add localize API to the ContainerManagementProtocol > --- > > Key: YARN-5557 > URL: https://issues.apache.org/jira/browse/YARN-5557 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5557.1.patch, YARN-5557.2.patch, YARN-5557.3.patch, > YARN-5557.4.patch > > > A new localize API for localizing new resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[
https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438922#comment-15438922
]
Rohith Sharma K S commented on YARN-3940:
-
+1LGTM
> Application moveToQueue should check NodeLabel permission
> --
>
> Key: YARN-3940
> URL: https://issues.apache.org/jira/browse/YARN-3940
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch,
> 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch,
> 0006-YARN-3940.patch, YARN-3940.0007.patch, YARN-3940.0008.patch,
> YARN-3940.0009.patch
>
>
> Configure capacity scheduler
> Configure node label an submit application {{queue=A Label=X}}
> Move application to queue {{B}} and x is not having access
> {code}
> 2015-07-20 19:46:19,626 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Application attempt appattempt_1437385548409_0005_01 released container
> container_e08_1437385548409_0005_01_02 on node: host:
> host-10-19-92-117:64318 #containers=1 available=
> used= with event: KILL
> 2015-07-20 19:46:20,970 WARN
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Invalid resource ask by application appattempt_1437385548409_0005_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, queue=b1 doesn't have permission to access all labels in
> resource request. labelExpression of resource request=x. Queue labels=y
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168)
> {code}
> Same exception will be thrown till *heartbeat timeout*
> Then application state will be updated to *FAILED*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5547) NMLeveldbStateStore should be more tolerant of unknown keys
[ https://issues.apache.org/jira/browse/YARN-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438921#comment-15438921 ] Jason Lowe commented on YARN-5547: -- Thanks for the patch! What I meant about the leak is a scenario like this: # NM is running version V which introduced a new key K that is associated with containers. # A container is running which causes K to be written to the state store # User does a rolling downgrade to V-1. The code ignores unrecognized key K. # The container completes and the container is removed from the state store. This only removes the container keys version V knows about, and K is not one of those keys. # At this point K has been leaked in the state store. # That leak will be permanent until a rolling upgrade to >= V. Even then K might not be cleaned up since all the other container state has been removed, probably interfering with the typical recovery flow for that key type. There are a couple of risks when cleaning up unrecognized keys. The old version may be removing the key too early in the lifecycle of that state such that if we do a rolling upgrade back to the version that works with those keys we've incorrectly destroyed the state. We probably need to think more about the ramifications of cleaning unrecognized keys and when we should or shouldn't do so. Appreciate any thoughts on this. The other risk is that doing this cleaning will add a place where the NM will read the state store as it scans for keys to remove, and previously it only ever wrote to the store after the initial recover on startup. Writes to leveldb are typically very fast, whereas reads could be much slower depending upon how much the database needs to be compacted and how many blocks are involved in the scan. This is likely a minor concern especially with the recent periodic full compaction to the store, but it will impact state store performance to some degree. As for the patch the changes will make the NM more tolerant of new container keys, but there are other places where unexpected keys will break the state store recovery. loadResourceTrackerState and loadUserLocalizedResources are some other places that should be updated and there are similar questions there as to what should be done about cleanup of unexpected keys. > NMLeveldbStateStore should be more tolerant of unknown keys > --- > > Key: YARN-5547 > URL: https://issues.apache.org/jira/browse/YARN-5547 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Ajith S > Attachments: YARN-5547.01.patch > > > Whenever new keys are added to the NM state store it will break rolling > downgrades because the code will throw if it encounters an unrecognized key. > If instead it skipped unrecognized keys it could be simpler to continue > supporting rolling downgrades. We need to define the semantics of > unrecognized keys when containers and apps are cleaned up, e.g.: we may want > to delete all keys underneath an app or container directory when it is being > removed from the state store to prevent leaking unrecognized keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4202) TestYarnClient#testReservationAPIs fails intermittently
[
https://issues.apache.org/jira/browse/YARN-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438866#comment-15438866
]
Ajith S commented on YARN-4202:
---
Found that this issue is fixed by YARN-4686
> TestYarnClient#testReservationAPIs fails intermittently
> ---
>
> Key: YARN-4202
> URL: https://issues.apache.org/jira/browse/YARN-4202
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Mit Desai
>Assignee: Ajith S
>
> Found this failure while looking at the Pre-run on one of my Jiras.
> {noformat}
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The planning algorithm could not find a valid allocation for your request
> at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1149)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:428)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:465)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2230)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2224)
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The planning algorithm could not find a valid allocation for your request
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.allocateUser(PlanningAlgorithm.java:69)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.createReservation(PlanningAlgorithm.java:140)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.TryManyReservationAgents.createReservation(TryManyReservationAgents.java:55)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy.createReservation(AlignedPlannerWithGreedy.java:84)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1132)
> ... 10 more
> {noformat}
> TestReport Link:
> https://builds.apache.org/job/PreCommit-YARN-Build/9243/testReport/
> When I ran this on my local box branch-2, it succeeds.
> {noformat}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.999 sec -
> in org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> Results :
> Tests run: 21, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 52.029 s
> [INFO] Finished at: 2015-09-23T11:25:04-06:00
> [INFO] Final Memory: 31M/391M
> [INFO]
>
> {noformat}
> Haven't tried if it is a problem in branch-2.7 or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5430) Return container's ip and host from NM ContainerStatus call
[ https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438813#comment-15438813 ] Hudson commented on YARN-5430: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10352 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10352/]) YARN-5430. Return container's ip and host from NM ContainerStatus call. (vvasudev: rev bfb4d950596d6198e473045dac6eb36c685b0319) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/runtime/ContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/Container.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DefaultLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DelegatingLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/api/protocolrecords/TestProtocolRecords.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerInspectCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerInspectCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/MockContainer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerStatus.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerStatusPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java > Return container's ip and host from NM ContainerStatus call > --- > > Key: YARN-5430 > URL: https://issues.apache.org/jira/browse/YARN-5430 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.9.0 > > Attachments: YARN-5430-branch-2.patch, YARN-5430.1.patch, > YARN-5430.2.patch, YARN-5430.3.patch, YARN-5430.4.patch, YARN-5430.5.patch, > YARN-5430.6.patch, YARN-5430.7.patch, YARN-5430.8.patch, > YARN-5430.9.branch-2.patch, YARN-5430.9.patch > > > In YARN-4757, we introduced a DNS mechanism for containers. That's based on > the assumption that we can get the container's ip and host information and > store it in the registry-service. This jira aims to get the container's ip > and host from the NM, primarily docker container -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-4202) TestYarnClient#testReservationAPIs fails intermittently
[
https://issues.apache.org/jira/browse/YARN-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajith S reassigned YARN-4202:
-
Assignee: Ajith S (was: nijel)
> TestYarnClient#testReservationAPIs fails intermittently
> ---
>
> Key: YARN-4202
> URL: https://issues.apache.org/jira/browse/YARN-4202
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Mit Desai
>Assignee: Ajith S
>
> Found this failure while looking at the Pre-run on one of my Jiras.
> {noformat}
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The planning algorithm could not find a valid allocation for your request
> at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1149)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:428)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:465)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2230)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2224)
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The planning algorithm could not find a valid allocation for your request
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.allocateUser(PlanningAlgorithm.java:69)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.createReservation(PlanningAlgorithm.java:140)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.TryManyReservationAgents.createReservation(TryManyReservationAgents.java:55)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy.createReservation(AlignedPlannerWithGreedy.java:84)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1132)
> ... 10 more
> {noformat}
> TestReport Link:
> https://builds.apache.org/job/PreCommit-YARN-Build/9243/testReport/
> When I ran this on my local box branch-2, it succeeds.
> {noformat}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.999 sec -
> in org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> Results :
> Tests run: 21, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 52.029 s
> [INFO] Finished at: 2015-09-23T11:25:04-06:00
> [INFO] Final Memory: 31M/391M
> [INFO]
>
> {noformat}
> Haven't tried if it is a problem in branch-2.7 or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4202) TestYarnClient#testReservationAPIs fails intermittently
[
https://issues.apache.org/jira/browse/YARN-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438806#comment-15438806
]
Ajith S commented on YARN-4202:
---
Hi [~nijel]
Would like to work on this, in case you are working on this, please feel free
to assign back
> TestYarnClient#testReservationAPIs fails intermittently
> ---
>
> Key: YARN-4202
> URL: https://issues.apache.org/jira/browse/YARN-4202
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Mit Desai
>Assignee: nijel
>
> Found this failure while looking at the Pre-run on one of my Jiras.
> {noformat}
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The planning algorithm could not find a valid allocation for your request
> at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1149)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:428)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:465)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2230)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2224)
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The planning algorithm could not find a valid allocation for your request
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.allocateUser(PlanningAlgorithm.java:69)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.createReservation(PlanningAlgorithm.java:140)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.TryManyReservationAgents.createReservation(TryManyReservationAgents.java:55)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy.createReservation(AlignedPlannerWithGreedy.java:84)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1132)
> ... 10 more
> {noformat}
> TestReport Link:
> https://builds.apache.org/job/PreCommit-YARN-Build/9243/testReport/
> When I ran this on my local box branch-2, it succeeds.
> {noformat}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.999 sec -
> in org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> Results :
> Tests run: 21, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 52.029 s
> [INFO] Finished at: 2015-09-23T11:25:04-06:00
> [INFO] Final Memory: 31M/391M
> [INFO]
>
> {noformat}
> Haven't tried if it is a problem in branch-2.7 or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5547) NMLeveldbStateStore should be more tolerant of unknown keys
[
https://issues.apache.org/jira/browse/YARN-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438766#comment-15438766
]
Ajith S commented on YARN-5547:
---
Hi [~jlowe]
I have attached the patch for handling the exception but i have a doubt about
{{prevent leaking unrecognized keys.}} . can you please elaborate.?
> NMLeveldbStateStore should be more tolerant of unknown keys
> ---
>
> Key: YARN-5547
> URL: https://issues.apache.org/jira/browse/YARN-5547
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Ajith S
> Attachments: YARN-5547.01.patch
>
>
> Whenever new keys are added to the NM state store it will break rolling
> downgrades because the code will throw if it encounters an unrecognized key.
> If instead it skipped unrecognized keys it could be simpler to continue
> supporting rolling downgrades. We need to define the semantics of
> unrecognized keys when containers and apps are cleaned up, e.g.: we may want
> to delete all keys underneath an app or container directory when it is being
> removed from the state store to prevent leaking unrecognized keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5547) NMLeveldbStateStore should be more tolerant of unknown keys
[ https://issues.apache.org/jira/browse/YARN-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajith S updated YARN-5547: -- Attachment: YARN-5547.01.patch > NMLeveldbStateStore should be more tolerant of unknown keys > --- > > Key: YARN-5547 > URL: https://issues.apache.org/jira/browse/YARN-5547 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Ajith S > Attachments: YARN-5547.01.patch > > > Whenever new keys are added to the NM state store it will break rolling > downgrades because the code will throw if it encounters an unrecognized key. > If instead it skipped unrecognized keys it could be simpler to continue > supporting rolling downgrades. We need to define the semantics of > unrecognized keys when containers and apps are cleaned up, e.g.: we may want > to delete all keys underneath an app or container directory when it is being > removed from the state store to prevent leaking unrecognized keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-2774) shared cache service should authorize calls properly
[ https://issues.apache.org/jira/browse/YARN-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438727#comment-15438727 ] Andrew Duffy edited comment on YARN-2774 at 8/26/16 9:50 AM: - It looks like all the dependent tickets have been closed out, is there anymore work going on here or can this be closed? EDIT: Never mind, looks like those were tickets for services that need authentication. was (Author: andreweduffy): It looks like all the dependent tickets have been closed out, is there anymore work going on here or can this be closed? > shared cache service should authorize calls properly > > > Key: YARN-2774 > URL: https://issues.apache.org/jira/browse/YARN-2774 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sangjin Lee > > The shared cache manager (SCM) services should authorize calls properly. > Currently, the uploader service (done in YARN-2186) does not authorize calls > to notify the SCM on newly uploaded resource. Proper security/authorization > needs to be done in this RPC call. Also, the use/release calls (done in > YARN-2188) and the scmAdmin commands (done in YARN-2189) are not properly > authorized. The SCM UI done in YARN-2203 as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2774) shared cache service should authorize calls properly
[ https://issues.apache.org/jira/browse/YARN-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438727#comment-15438727 ] Andrew Duffy commented on YARN-2774: It looks like all the dependent tickets have been closed out, is there anymore work going on here or can this be closed? > shared cache service should authorize calls properly > > > Key: YARN-2774 > URL: https://issues.apache.org/jira/browse/YARN-2774 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sangjin Lee > > The shared cache manager (SCM) services should authorize calls properly. > Currently, the uploader service (done in YARN-2186) does not authorize calls > to notify the SCM on newly uploaded resource. Proper security/authorization > needs to be done in this RPC call. Also, the use/release calls (done in > YARN-2188) and the scmAdmin commands (done in YARN-2189) are not properly > authorized. The SCM UI done in YARN-2203 as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5566) client-side NM graceful decom doesn't trigger when jobs finish
[
https://issues.apache.org/jira/browse/YARN-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438699#comment-15438699
]
Hadoop QA commented on YARN-5566:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 3s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m 28s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 39s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12825630/YARN-5566.001.patch |
| JIRA Issue | YARN-5566 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ecb9b6923d9d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 27c3b86 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12903/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12903/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> client-side NM graceful decom doesn't trigger when jobs finish
> --
>
> Key: YARN-5566
> URL: https://issues.apache.org/jira/browse/YARN-5566
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter:
[jira] [Updated] (YARN-5566) client-side NM graceful decom doesn't trigger when jobs finish
[
https://issues.apache.org/jira/browse/YARN-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated YARN-5566:
Attachment: YARN-5566.001.patch
When digging into this, I figured that the DECOMMISSIONG node must have thought
it was still running apps, and so I added a bunch of extra print statements and
saw that this was the case. While all RUNNING nodes had the correct counts,
the DECOMMISSIONING node's count went back up somehow. I was looking at the
state transitions, and saw that this one
{code:java}
.addTransition(NodeState.DECOMMISSIONING,
EnumSet.of(NodeState.DECOMMISSIONING, NodeState.DECOMMISSIONED),
RMNodeEventType.STATUS_UPDATE,
new StatusUpdateWhenHealthyTransition())
{code}
looked different from this analogue one
{code:java}
.addTransition(NodeState.RUNNING,
EnumSet.of(NodeState.RUNNING, NodeState.UNHEALTHY),
RMNodeEventType.STATUS_UPDATE,
new StatusUpdateWhenHealthyTransition())
{code}
despite calling the same transition. So I tried adding the UNHEALTHY state,
and that fixed the problem.
[~djp] any ideas what's going on here?
> client-side NM graceful decom doesn't trigger when jobs finish
> --
>
> Key: YARN-5566
> URL: https://issues.apache.org/jira/browse/YARN-5566
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-5566.001.patch
>
>
> I was testing the client-side NM graceful decommission and noticed that it
> was always waiting for the timeout, even if all jobs running on that node (or
> even the cluster) had already finished.
> For example:
> # JobA is running with at least one container on NodeA
> # User runs client-side decom on NodeA at 5:00am with a timeout of 3 hours
> --> NodeA enters DECOMMISSIONING state
> # JobA finishes at 6:00am and there are no other jobs running on NodeA
> # User's client reaches the timeout at 8:00am, and forcibly decommissions
> NodeA
> NodeA should have decommissioned at 6:00am.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5566) client-side NM graceful decom doesn't trigger when jobs finish
Robert Kanter created YARN-5566: --- Summary: client-side NM graceful decom doesn't trigger when jobs finish Key: YARN-5566 URL: https://issues.apache.org/jira/browse/YARN-5566 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.8.0 Reporter: Robert Kanter Assignee: Robert Kanter I was testing the client-side NM graceful decommission and noticed that it was always waiting for the timeout, even if all jobs running on that node (or even the cluster) had already finished. For example: # JobA is running with at least one container on NodeA # User runs client-side decom on NodeA at 5:00am with a timeout of 3 hours --> NodeA enters DECOMMISSIONING state # JobA finishes at 6:00am and there are no other jobs running on NodeA # User's client reaches the timeout at 8:00am, and forcibly decommissions NodeA NodeA should have decommissioned at 6:00am. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5308) FairScheduler: Move continuous scheduling related tests to TestContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438624#comment-15438624 ] Kai Sasaki commented on YARN-5308: -- [~varun_saxena] I rebased the patch. Could you check this again please? > FairScheduler: Move continuous scheduling related tests to > TestContinuousScheduling > --- > > Key: YARN-5308 > URL: https://issues.apache.org/jira/browse/YARN-5308 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler, test >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Kai Sasaki > Labels: newbie > Attachments: YARN-5308.01.patch, YARN-5308.02.patch, > YARN-5308.03.patch, YARN-5308.04.patch, YARN-5308.05.patch > > > TestFairScheduler still has some tests on continuous scheduling. We should > move them to TestContinuousScheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
