[jira] [Updated] (YARN-5430) Get container's ip and host from NM
[ https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-5430: -- Attachment: YARN-5430.5.patch > Get container's ip and host from NM > --- > > Key: YARN-5430 > URL: https://issues.apache.org/jira/browse/YARN-5430 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5430.1.patch, YARN-5430.2.patch, YARN-5430.3.patch, > YARN-5430.4.patch, YARN-5430.5.patch > > > In YARN-4757, we introduced a DNS mechanism for containers. That's based on > the assumption that we can get the container's ip and host information and > store it in the registry-service. This jira aims to get the container's ip > and host from the NM, primarily docker container -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5430) Get container's ip and host from NM
[
https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411274#comment-15411274
]
Hadoop QA commented on YARN-5430:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 58s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 57s
{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 57s {color} |
{color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 57s {color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 8
new + 166 unchanged - 2 fixed = 174 total (was 168) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 18s {color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 18s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.logaggregation.TestAggregatedLogFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822504/YARN-5430.4.patch |
| JIRA Issue | YARN-5430 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 16a8dd1f6d96 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision |
[jira] [Updated] (YARN-5430) Get container's ip and host from NM
[ https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-5430: -- Attachment: (was: YARN-5430.4.patch) > Get container's ip and host from NM > --- > > Key: YARN-5430 > URL: https://issues.apache.org/jira/browse/YARN-5430 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5430.1.patch, YARN-5430.2.patch, YARN-5430.3.patch, > YARN-5430.4.patch > > > In YARN-4757, we introduced a DNS mechanism for containers. That's based on > the assumption that we can get the container's ip and host information and > store it in the registry-service. This jira aims to get the container's ip > and host from the NM, primarily docker container -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5430) Get container's ip and host from NM
[ https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-5430: -- Attachment: YARN-5430.4.patch > Get container's ip and host from NM > --- > > Key: YARN-5430 > URL: https://issues.apache.org/jira/browse/YARN-5430 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5430.1.patch, YARN-5430.2.patch, YARN-5430.3.patch, > YARN-5430.4.patch, YARN-5430.4.patch > > > In YARN-4757, we introduced a DNS mechanism for containers. That's based on > the assumption that we can get the container's ip and host information and > store it in the registry-service. This jira aims to get the container's ip > and host from the NM, primarily docker container -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5430) Get container's ip and host from NM
[
https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411234#comment-15411234
]
Hadoop QA commented on YARN-5430:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} YARN-5430 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822230/YARN-5430.4.patch |
| JIRA Issue | YARN-5430 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12666/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Get container's ip and host from NM
> ---
>
> Key: YARN-5430
> URL: https://issues.apache.org/jira/browse/YARN-5430
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5430.1.patch, YARN-5430.2.patch, YARN-5430.3.patch,
> YARN-5430.4.patch
>
>
> In YARN-4757, we introduced a DNS mechanism for containers. That's based on
> the assumption that we can get the container's ip and host information and
> store it in the registry-service. This jira aims to get the container's ip
> and host from the NM, primarily docker container
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411191#comment-15411191 ] Jian He commented on YARN-5382: --- lgtm, [~jlowe], would you like to check? > RM does not audit log kill request for active applications > -- > > Key: YARN-5382 > URL: https://issues.apache.org/jira/browse/YARN-5382 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Vrushali C > Attachments: YARN-5382-branch-2.7.01.patch, > YARN-5382-branch-2.7.02.patch, YARN-5382-branch-2.7.03.patch, > YARN-5382-branch-2.7.04.patch, YARN-5382-branch-2.7.05.patch, > YARN-5382-branch-2.7.09.patch, YARN-5382-branch-2.7.10.patch, > YARN-5382-branch-2.7.11.patch, YARN-5382-branch-2.7.12.patch, > YARN-5382.06.patch, YARN-5382.07.patch, YARN-5382.08.patch, > YARN-5382.09.patch, YARN-5382.10.patch, YARN-5382.11.patch, > YARN-5382.12.patch, YARN-5382.13.patch > > > ClientRMService will audit a kill request but only if it either fails to > issue the kill or if the kill is sent to an already finished application. It > does not create a log entry when the application is active which is arguably > the most important case to audit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5479) FairScheduler: Scheduling performance improvement
[
https://issues.apache.org/jira/browse/YARN-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411121#comment-15411121
]
Ray Chiang commented on YARN-5479:
--
What you have in mind sounds interesting. I'd have to look at parts of the
codebase more to comment further, but just some food for thought:
- I'd be careful with having multiple implementations or multiple APIs for
doing the same thing with Resource. Resource is used a lot of places in the
Hadoop codebase and this could add confusion, even with accurate Javadocs.
- The nodeUpdate() changes will conflict with YARN-5047 unless you plan on
doing the same changes for CapacityScheduler and FifoScheduler.
> FairScheduler: Scheduling performance improvement
> -
>
> Key: YARN-5479
> URL: https://issues.apache.org/jira/browse/YARN-5479
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, resourcemanager
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>Assignee: He Tianyi
>
> Currently ResourceManager uses a single thread to handle async events for
> scheduling. As number of nodes grows, more events need to be processed in
> time in FairScheduler. Also, increased number of applications & queues slows
> down processing of each single event.
> There are two cases that slow processing of nodeUpdate events is problematic:
> A. global throughput is lower than number of nodes through heartbeat rounds.
> This keeps resource from being allocated since the inefficiency.
> B. global throughput meets the need, but for some of these rounds, events of
> some nodes cannot get processed before next heartbeat. This brings
> inefficiency handling burst requests (i.e. newly submitted MapReduce
> application cannot get its all task launched soon given enough resource).
> Pretty sure some people will encounter the problem eventually after a single
> cluster is scaled to several K of nodes (even with {{assignmultiple}}
> enabled).
> This issue proposes to perform several optimization towards performance in
> FairScheduler {{nodeUpdate}} method. To be specific:
> A. trading off fairness with efficiency, queue & app sorting can be skipped
> (or should this be called 'delayed sorting'?). we can either start another
> dedicated thread to do the sorting & updating, or actually perform sorting
> after current result have been used several times (say sort once in every 100
> calls.)
> B. performing calculation on {{Resource}} instances is expensive, since at
> least 2 objects ({{ResourceImpl}} and its proto builder) is created each time
> (using 'immutable' apis). the overhead can be eliminated with a
> light-weighted implementation of Resource, which do not instantiate a builder
> until necessary, because most instances are used as intermediate result in
> scheduler instead of being exchanged via IPC. Also, {{createResource}} is
> using reflection, which can be replaced by a plain {{new}} (for scheduler
> usage only). furthermore, perhaps we could 'intern' resource to avoid
> allocation.
> C. other minor changes: such as move {{updateRootMetrics}} call to
> {{update}}, making root queue metrics eventual consistent (which may
> satisfies most of the needs). or introduce counters to {{getResourceUsage}}
> and make changing of resource incrementally instead of recalculate each time.
> With A and B, I was looking at 4 times improvement in a cluster with 2K nodes.
> Suggestions? Opinions?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5474) Typo mistake in AMRMClient#getRegisteredTimeineClient API
[
https://issues.apache.org/jira/browse/YARN-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411049#comment-15411049
]
Hadoop QA commented on YARN-5474:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 18s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 42s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822480/YARN-5474.v1.001.patch
|
| JIRA Issue | YARN-5474 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 178d8d72fe73 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 131d58a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12665/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12665/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12665/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12665/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Typo mistake in AMRMClient#getRegisteredTimeineClient API
>
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[
https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411043#comment-15411043
]
Hadoop QA commented on YARN-5287:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 42s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m 58s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.8.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 46s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822477/YARN-5287.branch-2.001.patch
|
| JIRA Issue | YARN-5287 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 88905281c52d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / 9cdd97a |
| Default Java | 1.7.0_101 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_101
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 |
| JDK v1.7.0_101 Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12664/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12664/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> LinuxContainerExecutor fails to set proper permission
> -
>
> Key: YARN-5287
> URL: https://issues.apache.org/jira/browse/YARN-5287
> Project: Hadoop YARN
> Issue Type: Bug
>
[jira] [Updated] (YARN-5474) Typo mistake in AMRMClient#getRegisteredTimeineClient API
[
https://issues.apache.org/jira/browse/YARN-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-5474:
Attachment: YARN-5474.v1.001.patch
attached a patch, IMO as its just a method signature change we do not req a
test case
> Typo mistake in AMRMClient#getRegisteredTimeineClient API
> -
>
> Key: YARN-5474
> URL: https://issues.apache.org/jira/browse/YARN-5474
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Naganarasimha G R
>Priority: Trivial
> Labels: newbie
> Attachments: YARN-5474.v1.001.patch
>
>
> Just found that typo mistake in the API, It can be fixed since ATS is not
> release in any version.
> {code}
> /**
>* Get registered timeline client.
>* @return the registered timeline client
>*/
> public TimelineClient getRegisteredTimeineClient() {
> return this.timelineClient;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410915#comment-15410915 ] Naganarasimha G R edited comment on YARN-5287 at 8/7/16 6:40 PM: - Thanks for the contributions [~Ying Zhang] and review by [~rohithsharma] & [~vvasudev], Seems like patch fails to apply on Branch-2, currently working on re basing the patch for branch-2. Was also planning to commit this in 2.8 too ... was (Author: naganarasimha): Thanks for the contributions [~Ying Zhang] and review by [~rohithsharma] & [~vvasudev], Seems like patch fails to apply on Branch-2, currently working on re basing the patch for 2.8. Was also planning to commit this in 2.8 too ... > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch, YARN-5287.005.patch, YARN-5287.branch-2.001.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-5287: Attachment: YARN-5287.branch-2.001.patch hi [~Ying Zhang], Can you review this patch for branch-2 ? > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch, YARN-5287.005.patch, YARN-5287.branch-2.001.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5481) Add Aggregate scheduler counts as part of scheduler Rest API
Bibin A Chundatt created YARN-5481: -- Summary: Add Aggregate scheduler counts as part of scheduler Rest API Key: YARN-5481 URL: https://issues.apache.org/jira/browse/YARN-5481 Project: Hadoop YARN Issue Type: Improvement Reporter: Bibin A Chundatt Priority: Minor Currently the aggregate scheduler count are missing as part for Scheduler Rest API.Add the same as part Rest API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5481) Add Aggregate scheduler counts as part of scheduler Rest API
[ https://issues.apache.org/jira/browse/YARN-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt reassigned YARN-5481: -- Assignee: Bibin A Chundatt > Add Aggregate scheduler counts as part of scheduler Rest API > - > > Key: YARN-5481 > URL: https://issues.apache.org/jira/browse/YARN-5481 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > > Currently the aggregate scheduler count are missing as part for Scheduler > Rest API.Add the same as part Rest API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5480) Add detail of IncrementResource in RM Web UI
Bibin A Chundatt created YARN-5480:
--
Summary: Add detail of IncrementResource in RM Web UI
Key: YARN-5480
URL: https://issues.apache.org/jira/browse/YARN-5480
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Default minimum allocation in capacity scheduler is {{1024}} and parameter
considered for incrementResource while Normalize request is minimum scheduler
capacity.
Currently with default configuration submit mapreduce application with default
am memory {{1536}} the AM resource allocated will be {{2048}}.
Which creates confusion even when memory for request is more than {{1024}} the
why the allocation is not {{1536}}.
>From RM web UI only 2 parameters will able to understand minimum and max value
>for scheduler allocation.
Would like to handle as below.
# Add increment param as part of RM Web UI
# Update documentation to explain.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[ https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411014#comment-15411014 ] Junping Du commented on YARN-4676: -- Move target version to 2.9 as our previous agreement that we need to handle RM restart case with this patch in the same release. > Automatic and Asynchronous Decommissioning Nodes Status Tracking > > > Key: YARN-4676 > URL: https://issues.apache.org/jira/browse/YARN-4676 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daniel Zhi >Assignee: Daniel Zhi > Labels: features > Attachments: GracefulDecommissionYarnNode.pdf, > GracefulDecommissionYarnNode.pdf, YARN-4676.004.patch, YARN-4676.005.patch, > YARN-4676.006.patch, YARN-4676.007.patch, YARN-4676.008.patch, > YARN-4676.009.patch, YARN-4676.010.patch, YARN-4676.011.patch, > YARN-4676.012.patch, YARN-4676.013.patch, YARN-4676.014.patch, > YARN-4676.015.patch, YARN-4676.016.patch, YARN-4676.017.patch, > YARN-4676.018.patch, YARN-4676.019.patch, YARN-4676.020.patch > > > YARN-4676 implements an automatic, asynchronous and flexible mechanism to > graceful decommission > YARN nodes. After user issues the refreshNodes request, ResourceManager > automatically evaluates > status of all affected nodes to kicks out decommission or recommission > actions. RM asynchronously > tracks container and application status related to DECOMMISSIONING nodes to > decommission the > nodes immediately after there are ready to be decommissioned. Decommissioning > timeout at individual > nodes granularity is supported and could be dynamically updated. The > mechanism naturally supports multiple > independent graceful decommissioning “sessions” where each one involves > different sets of nodes with > different timeout settings. Such support is ideal and necessary for graceful > decommission request issued > by external cluster management software instead of human. > DecommissioningNodeWatcher inside ResourceTrackingService tracks > DECOMMISSIONING nodes status automatically and asynchronously after > client/admin made the graceful decommission request. It tracks > DECOMMISSIONING nodes status to decide when, after all running containers on > the node have completed, will be transitioned into DECOMMISSIONED state. > NodesListManager detect and handle include and exclude list changes to kick > out decommission or recommission as necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[ https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-4676: - Target Version/s: 2.9.0 (was: 2.8.0) > Automatic and Asynchronous Decommissioning Nodes Status Tracking > > > Key: YARN-4676 > URL: https://issues.apache.org/jira/browse/YARN-4676 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daniel Zhi >Assignee: Daniel Zhi > Labels: features > Attachments: GracefulDecommissionYarnNode.pdf, > GracefulDecommissionYarnNode.pdf, YARN-4676.004.patch, YARN-4676.005.patch, > YARN-4676.006.patch, YARN-4676.007.patch, YARN-4676.008.patch, > YARN-4676.009.patch, YARN-4676.010.patch, YARN-4676.011.patch, > YARN-4676.012.patch, YARN-4676.013.patch, YARN-4676.014.patch, > YARN-4676.015.patch, YARN-4676.016.patch, YARN-4676.017.patch, > YARN-4676.018.patch, YARN-4676.019.patch, YARN-4676.020.patch > > > YARN-4676 implements an automatic, asynchronous and flexible mechanism to > graceful decommission > YARN nodes. After user issues the refreshNodes request, ResourceManager > automatically evaluates > status of all affected nodes to kicks out decommission or recommission > actions. RM asynchronously > tracks container and application status related to DECOMMISSIONING nodes to > decommission the > nodes immediately after there are ready to be decommissioned. Decommissioning > timeout at individual > nodes granularity is supported and could be dynamically updated. The > mechanism naturally supports multiple > independent graceful decommissioning “sessions” where each one involves > different sets of nodes with > different timeout settings. Such support is ideal and necessary for graceful > decommission request issued > by external cluster management software instead of human. > DecommissioningNodeWatcher inside ResourceTrackingService tracks > DECOMMISSIONING nodes status automatically and asynchronously after > client/admin made the graceful decommission request. It tracks > DECOMMISSIONING nodes status to decide when, after all running containers on > the node have completed, will be transitioned into DECOMMISSIONED state. > NodesListManager detect and handle include and exclude list changes to kick > out decommission or recommission as necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[
https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411012#comment-15411012
]
Junping Du commented on YARN-4676:
--
Thanks for updating the patch, [~danzhi]! I went through the whole patch this
weekend except the documentation part. I would suggest to separate document
code into a separated JIRA if we don't want this JIRA lingering on for longer
time as reviewing the doc is really a bit difficult in general.
My overall feeling for the patch is we are getting closer, but still many
issues to address, see below:
In HostsFileReader.java,
{noformat}
+ public synchronized Map getExcludedHostsWithTimeout() {
+return excludes;
+ }
{noformat}
Two issues here: 1. synchronized doesn't work here if other places update
excludes (protected in write lock), we should use read lock to protected. 2. we
shouldn't return excludes directly which could cause leak of writer lock
because the caller can potentially update the map here. Please refer example
like getHostDetails() which is more thread safe in this case.
{noformat}
+ // Examples:
+ // host1
+ // host2123
+ // host3-1
{noformat}
Can we allow multiple hosts in the same item if their timeout value is the
same, like: "host1, host2,
host3123"? It sounds much concisely and can
save lots of duplicated meta info. Should be easy to address. Isn't it?
In RMAdminCLI.java,
{noformat}
+ public static void readXmlFileToMapWithFileInputStream(String type,
+ String filename, InputStream fileInputStream, Map map)
+ throws IOException
{noformat}
I don't see how we throw IOException here given IOException is already catched
and throw runtime exception instead. We should remove if it is not necessary.
{noformat}
+ if ("-g".equals(args[1]) || "-graceful".equals(args[1])) {
{noformat}
If we want to support "-graceful" in addition to -g, then we should mention it
in commandline usage messages also.
In DecommissioningNodesWatcher.java,
{noformat}
+ * DecommissioningNodesWatcher basically is no cost when no node is
+ * DECOMMISSIONING. This sacrifices the possibility that that node once
+ * host containers of an still running application.
{noformat}
I don't quite understand the last sentense here. Can you have more
clarification here?
In {{void update(RMNode rmNode, NodeStatus remoteNodeStatus)}},
{noformat}
+if (rmNode.getState() == NodeState.DECOMMISSIONED) {
+ if (context == null) {
+return;
+ }
+ context.nodeState = rmNode.getState();
+ // keep DECOMMISSIONED node for a while for status log.
+ if (context.decommissionedTime == 0) {
+context.decommissionedTime = now;
+ } else if (now - context.decommissionedTime > 6L) {
+decomNodes.remove(rmNode.getNodeID());
+LOG.info("remove " + rmNode.getState() + " " + rmNode.getNodeID());
+ }
{noformat}
Why we are waiting for 60 seconds to remove node from decomNodes after node in
DECOMMISSIONED? Shouldn't we remove it directly?
{noformat}
+long waitTime =
+System.currentTimeMillis() - context.decommissioningStartTime;
{noformat}
We should use MonotonicClock to track all timeout calculations which won't
affected by system clock changes, please refer AbstractLivelinessMonitor as an
example.
PollTimerTask.run()
The logic here need more improvements: 1. it doesn't have any sleep time for
monitoring interval - I don't think we should so tightly loop here; 2. we don't
check the NM that updated in latest 30 seconds or 60 seconds which sounds too
gross-grained timeout tracking. 3. Many unnecessary log operations.
I think we should follow similar logic in AbstractLivelinessMonitor although we
cannot use it directly as each node could have different expire interval. Also,
we should move the whole logic to loop staleNodes to check timeout/ready and
send DECOMMISSION and a statemachine transition that can be triggered directly
when received a READY or TIMEOUT event.
I will do more tests in next 1 day or 2 to make sure it won't affect forcefully
decommission functionality and client side timeout tracking.
> Automatic and Asynchronous Decommissioning Nodes Status Tracking
>
>
> Key: YARN-4676
> URL: https://issues.apache.org/jira/browse/YARN-4676
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Daniel Zhi
>Assignee: Daniel Zhi
> Labels: features
> Attachments: GracefulDecommissionYarnNode.pdf,
> GracefulDecommissionYarnNode.pdf, YARN-4676.004.patch, YARN-4676.005.patch,
> YARN-4676.006.patch, YARN-4676.007.patch, YARN-4676.008.patch,
> YARN-4676.009.patch, YARN-4676.010.patch, YARN-4676.011.patch,
>
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410915#comment-15410915 ] Naganarasimha G R commented on YARN-5287: - Thanks for the contributions [~Ying Zhang] and review by [~rohithsharma] & [~vvasudev], Seems like patch fails to apply on Branch-2, currently working on re basing the patch for 2.8. Was also planning to commit this in 2.8 too ... > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch, YARN-5287.005.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
