[jira] [Updated] (YARN-6061) Add a customized uncaughtexceptionhandler for critical threads in RM
[ https://issues.apache.org/jira/browse/YARN-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6061: --- Attachment: YARN-6061.009.patch > Add a customized uncaughtexceptionhandler for critical threads in RM > > > Key: YARN-6061 > URL: https://issues.apache.org/jira/browse/YARN-6061 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6061.001.patch, YARN-6061.002.patch, > YARN-6061.003.patch, YARN-6061.004.patch, YARN-6061.005.patch, > YARN-6061.006.patch, YARN-6061.007.patch, YARN-6061.008.patch, > YARN-6061.009.patch > > > There are several threads in fair scheduler. The thread will quit when there > is a runtime exception inside it. We should bring down the RM when that > happens. Otherwise, there may be some weird behavior in RM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860857#comment-15860857
]
Hadoop QA commented on YARN-6163:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 104 unchanged - 1 fixed = 109 total (was 105) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 42s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12851992/yarn-6163-1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 51c36a03a66d 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 08f9397 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14877/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14877/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
[jira] [Commented] (YARN-6151) FS preemption does not consider child queues over fairshare if the parent is under
[
https://issues.apache.org/jira/browse/YARN-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860856#comment-15860856
]
Yufei Gu commented on YARN-6151:
Thanks [~kasha] for the review and commit.
> FS preemption does not consider child queues over fairshare if the parent is
> under
> --
>
> Key: YARN-6151
> URL: https://issues.apache.org/jira/browse/YARN-6151
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.8.0
>
> Attachments: YARN-6151.branch-2.8.001.patch,
> YARN-6151.branch-2.8.002.patch, YARN-6151.branch-2.8.003.patch
>
>
> This is preemption bug happens before 2.8.0, which also described in
> YARN-3405.
> Queue hierarchy described as below:
> {noformat}
> root
>/ \
>queue-1 queue-2
> / \
> queue-1-1 queue-1-2
> {noformat}
> Assume cluster resource is 100 and all queues have same weights.
> # queue-1-1 and queue-2 has apps. Each get 50 usage and 50 fairshare.
> # When queue-1-2 is active, supposedly it will preempt 25 from queue-1-1, but
> this doesn't happen because preemption happens top-down, queue-2 could be the
> preemption candidate as long as queue-2 is less needy than queue-1, and
> queue-2 doesn't exceed the fair share which means preemption won't happen.
> We need to filter out queue-2 since it isn't a valid candidate.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[
https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860854#comment-15860854
]
Hadoop QA commented on YARN-6163:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 106 unchanged - 1 fixed = 111 total (was 107) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
53s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 42m 25s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12851992/yarn-6163-1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fd0282598150 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 08f9397 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14876/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14876/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
[jira] [Commented] (YARN-5889) Improve and refactor user-limit calculation in capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860851#comment-15860851 ] Sunil G commented on YARN-5889: --- Thanks for thorough reviews and commit [~leftnoteasy] , and thanks for the detailed review from [~eepayne] and [~jlowe]. Really appreciate the same. I am working on user-limit preemption poc based on this patch and will shortly upload in preemption jira. > Improve and refactor user-limit calculation in capacity scheduler > - > > Key: YARN-5889 > URL: https://issues.apache.org/jira/browse/YARN-5889 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5889.0001.patch, > YARN-5889.0001.suggested.patchnotes, YARN-5889.0002.patch, > YARN-5889.0003.patch, YARN-5889.0004.patch, YARN-5889.0005.patch, > YARN-5889.0006.patch, YARN-5889.0007.patch, YARN-5889.0008.patch, > YARN-5889.0009.patch, YARN-5889.0010.patch, YARN-5889.v0.patch, > YARN-5889.v1.patch, YARN-5889.v2.patch > > > Currently user-limit is computed during every heartbeat allocation cycle with > a write lock. To improve performance, this tickets is focussing on moving > user-limit calculation out of heartbeat allocation flow. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2
Sangjin Lee created YARN-6170:
-
Summary: TimelineReaderServer should wait to join with HttpServer2
Key: YARN-6170
URL: https://issues.apache.org/jira/browse/YARN-6170
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelinereader
Affects Versions: YARN-5355
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Priority: Minor
While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I
noticed that the timeline reader daemon would promptly shut down upon start. It
turns out that in the 2.6.0 code line at least there are only daemon threads
left once the main method returns. That causes the JVM to shut down.
The right pattern to start an embedded jetty web server is to call
{{Server.start()}} followed by {{Server.join()}}. That way, the server stays up
reliably no matter what other threads get created.
It works on YARN-5355 only because there *happens* to be one other non-daemon
thread. We should add the {{join()}} call to be always correct.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Improve /flows API for more flexible filters fromid, collapse, userid
[ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860820#comment-15860820 ] Rohith Sharma K S commented on YARN-6027: - Thanks [~sjlee0] for summarizing! Could you comment more about the reason for not agreeing collapse filter? (Though I was there in call, I would request other folks to add it). I will create a new JIRA for discussion more on collapse filter support. > Improve /flows API for more flexible filters fromid, collapse, userid > - > > Key: YARN-6027 > URL: https://issues.apache.org/jira/browse/YARN-6027 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6027-YARN-5355.0001.patch > > > In YARN-5585 , fromId is supported for retrieving entities. We need similar > filter for flows/flowRun apps and flow run and flow as well. > Along with supporting fromId, this JIRA should also discuss following points > * Should we throw an exception for entities/entity retrieval if duplicates > found? > * TimelieEntity : > ** Should equals method also check for idPrefix? > ** Does idPrefix is part of identifiers? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5271) ATS client doesn't work with Jersey 2 on the classpath
[ https://issues.apache.org/jira/browse/YARN-5271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860760#comment-15860760 ] Weiwei Yang commented on YARN-5271: --- Thanks [~gtCarrera9], hi [~jojochuang] what's your thought on this? > ATS client doesn't work with Jersey 2 on the classpath > -- > > Key: YARN-5271 > URL: https://issues.apache.org/jira/browse/YARN-5271 > Project: Hadoop YARN > Issue Type: Bug > Components: client, timelineserver >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Weiwei Yang > Labels: oct16-medium > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: YARN-5271.01.patch, YARN-5271.02.patch, > YARN-5271.branch-2.01.patch, YARN-5271-branch-2.8.01.patch > > > see SPARK-15343 : once Jersey 2 is on the CP, you can't instantiate a > timeline client, *even if the server is an ATS1.5 server and publishing is > via the FS* -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6163) FS Preemption is a trickle for severely starved applications
[ https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860750#comment-15860750 ] Karthik Kambatla commented on YARN-6163: Patch that considers multiple RRs for an app. Tracking individual containers was too complicated for little gain. > FS Preemption is a trickle for severely starved applications > > > Key: YARN-6163 > URL: https://issues.apache.org/jira/browse/YARN-6163 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-6163-1.patch > > > With current logic, only one RR is considered per each instance of marking an > application starved. This marking happens only on the update call that runs > every 500ms. Due to this, an application that is severely starved takes > forever to reach fairshare based on preemptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6163) FS Preemption is a trickle for severely starved applications
[ https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-6163: --- Attachment: yarn-6163-1.patch > FS Preemption is a trickle for severely starved applications > > > Key: YARN-6163 > URL: https://issues.apache.org/jira/browse/YARN-6163 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-6163-1.patch > > > With current logic, only one RR is considered per each instance of marking an > application starved. This marking happens only on the update call that runs > every 500ms. Due to this, an application that is severely starved takes > forever to reach fairshare based on preemptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5703) ReservationAgents are not correctly configured
[
https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860746#comment-15860746
]
Hadoop QA commented on YARN-5703:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 14 new + 94 unchanged - 0 fixed = 108 total (was 94)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 18 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
20s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 3 new + 908 unchanged - 0 fixed = 911 total (was 908) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 52s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5703 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12851427/YARN-5703.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux dc50c9155cd0 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 08f9397 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14875/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/14875/artifact/patchprocess/whitespace-eol.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/14875/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-6141) ppc64le on Linux doesn't trigger __linux get_executable codepath
[
https://issues.apache.org/jira/browse/YARN-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860687#comment-15860687
]
Sonia Garudi commented on YARN-6141:
On upgrading the version of cmake to >3.1 the build is successful. For cmake
version < 3.1, the C standard is being set to c99 in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt:
{code:borderStyle=solid}
if (CMAKE_VERSION VERSION_LESS "3.1")
# subset of CMAKE__COMPILER_ID
#
https://cmake.org/cmake/help/v3.0/variable/CMAKE_LANG_COMPILER_ID.html
if (CMAKE_C_COMPILER_ID STREQUAL "GNU" OR
CMAKE_C_COMPILER_ID STREQUAL "Clang" OR
CMAKE_C_COMPILER_ID STREQUAL "AppleClang")
set (CMAKE_C_FLAGS "-std=c99 -Wall -pedantic-errors
${CMAKE_C_FLAGS}")
elseif (CMAKE_C_COMPILER_ID STREQUAL "Intel")
set (CMAKE_C_FLAGS "-std=c99 -Wall ${CMAKE_C_FLAGS}")
elseif (CMAKE_C_COMPILER_ID STREQUAL "SunPro")
set (CMAKE_C_FLAGS "-xc99 ${CMAKE_C_FLAGS}")
endif ()
else ()
set (CMAKE_C_STANDARD 99)
endif ()
{code}
> ppc64le on Linux doesn't trigger __linux get_executable codepath
>
>
> Key: YARN-6141
> URL: https://issues.apache.org/jira/browse/YARN-6141
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha3
> Environment: $ uname -a
> Linux f8eef0f055cf 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Sonia Garudi
> Labels: ppc64le
> Attachments: YARN-6141.diff
>
>
> On ppc64le architecture, the build fails in the 'Hadoop YARN NodeManager'
> project with the below error:
> Cannot safely determine executable path with a relative HADOOP_CONF_DIR on
> this operating system.
> [WARNING] #error Cannot safely determine executable path with a relative
> HADOOP_CONF_DIR on this operating system.
> [WARNING] ^
> [WARNING] make[2]: ***
> [CMakeFiles/container.dir/main/native/container-executor/impl/get_executable.c.o]
> Error 1
> [WARNING] make[2]: *** Waiting for unfinished jobs
> [WARNING] make[1]: *** [CMakeFiles/container.dir/all] Error 2
> [WARNING] make: *** [all] Error 2
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> Cmake version used :
> $ /usr/bin/cmake --version
> cmake version 2.8.12.2
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6151) FS preemption does not consider child queues over fairshare if the parent is under
[
https://issues.apache.org/jira/browse/YARN-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860681#comment-15860681
]
Karthik Kambatla commented on YARN-6151:
+1. Just committed this to branch-2.8.
> FS preemption does not consider child queues over fairshare if the parent is
> under
> --
>
> Key: YARN-6151
> URL: https://issues.apache.org/jira/browse/YARN-6151
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6151.branch-2.8.001.patch,
> YARN-6151.branch-2.8.002.patch, YARN-6151.branch-2.8.003.patch
>
>
> This is preemption bug happens before 2.8.0, which also described in
> YARN-3405.
> Queue hierarchy described as below:
> {noformat}
> root
>/ \
>queue-1 queue-2
> / \
> queue-1-1 queue-1-2
> {noformat}
> Assume cluster resource is 100 and all queues have same weights.
> # queue-1-1 and queue-2 has apps. Each get 50 usage and 50 fairshare.
> # When queue-1-2 is active, supposedly it will preempt 25 from queue-1-1, but
> this doesn't happen because preemption happens top-down, queue-2 could be the
> preemption candidate as long as queue-2 is less needy than queue-1, and
> queue-2 doesn't exceed the fair share which means preemption won't happen.
> We need to filter out queue-2 since it isn't a valid candidate.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6151) FS preemption does not consider child queues over fairshare if the parent is under
[
https://issues.apache.org/jira/browse/YARN-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860682#comment-15860682
]
Karthik Kambatla commented on YARN-6151:
Thanks for fixing this, Yufei.
> FS preemption does not consider child queues over fairshare if the parent is
> under
> --
>
> Key: YARN-6151
> URL: https://issues.apache.org/jira/browse/YARN-6151
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6151.branch-2.8.001.patch,
> YARN-6151.branch-2.8.002.patch, YARN-6151.branch-2.8.003.patch
>
>
> This is preemption bug happens before 2.8.0, which also described in
> YARN-3405.
> Queue hierarchy described as below:
> {noformat}
> root
>/ \
>queue-1 queue-2
> / \
> queue-1-1 queue-1-2
> {noformat}
> Assume cluster resource is 100 and all queues have same weights.
> # queue-1-1 and queue-2 has apps. Each get 50 usage and 50 fairshare.
> # When queue-1-2 is active, supposedly it will preempt 25 from queue-1-1, but
> this doesn't happen because preemption happens top-down, queue-2 could be the
> preemption candidate as long as queue-2 is less needy than queue-1, and
> queue-2 doesn't exceed the fair share which means preemption won't happen.
> We need to filter out queue-2 since it isn't a valid candidate.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-6141) ppc64le on Linux doesn't trigger __linux get_executable codepath
[
https://issues.apache.org/jira/browse/YARN-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sonia Garudi updated YARN-6141:
---
Comment: was deleted
(was: On upgrading the version of cmake to >3.1 the build is successful. For
cmake version < 3.1, the C standard is being set to c99 in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt:
if (CMAKE_VERSION VERSION_LESS "3.1")
# subset of CMAKE__COMPILER_ID
#
https://cmake.org/cmake/help/v3.0/variable/CMAKE_LANG_COMPILER_ID.html
if (CMAKE_C_COMPILER_ID STREQUAL "GNU" OR
CMAKE_C_COMPILER_ID STREQUAL "Clang" OR
CMAKE_C_COMPILER_ID STREQUAL "AppleClang")
set (CMAKE_C_FLAGS "-std=c99 -Wall -pedantic-errors
${CMAKE_C_FLAGS}")
elseif (CMAKE_C_COMPILER_ID STREQUAL "Intel")
set (CMAKE_C_FLAGS "-std=c99 -Wall ${CMAKE_C_FLAGS}")
elseif (CMAKE_C_COMPILER_ID STREQUAL "SunPro")
set (CMAKE_C_FLAGS "-xc99 ${CMAKE_C_FLAGS}")
endif ()
else ()
set (CMAKE_C_STANDARD 99)
endif ()
)
> ppc64le on Linux doesn't trigger __linux get_executable codepath
>
>
> Key: YARN-6141
> URL: https://issues.apache.org/jira/browse/YARN-6141
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha3
> Environment: $ uname -a
> Linux f8eef0f055cf 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Sonia Garudi
> Labels: ppc64le
> Attachments: YARN-6141.diff
>
>
> On ppc64le architecture, the build fails in the 'Hadoop YARN NodeManager'
> project with the below error:
> Cannot safely determine executable path with a relative HADOOP_CONF_DIR on
> this operating system.
> [WARNING] #error Cannot safely determine executable path with a relative
> HADOOP_CONF_DIR on this operating system.
> [WARNING] ^
> [WARNING] make[2]: ***
> [CMakeFiles/container.dir/main/native/container-executor/impl/get_executable.c.o]
> Error 1
> [WARNING] make[2]: *** Waiting for unfinished jobs
> [WARNING] make[1]: *** [CMakeFiles/container.dir/all] Error 2
> [WARNING] make: *** [all] Error 2
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> Cmake version used :
> $ /usr/bin/cmake --version
> cmake version 2.8.12.2
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6151) FS preemption does not consider child queues over fairshare if the parent is under
[
https://issues.apache.org/jira/browse/YARN-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6151:
---
Summary: FS preemption does not consider child queues over fairshare if the
parent is under (was: FS Preemption doesn't filter out queues which cannot be
preempted)
> FS preemption does not consider child queues over fairshare if the parent is
> under
> --
>
> Key: YARN-6151
> URL: https://issues.apache.org/jira/browse/YARN-6151
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6151.branch-2.8.001.patch,
> YARN-6151.branch-2.8.002.patch, YARN-6151.branch-2.8.003.patch
>
>
> This is preemption bug happens before 2.8.0, which also described in
> YARN-3405.
> Queue hierarchy described as below:
> {noformat}
> root
>/ \
>queue-1 queue-2
> / \
> queue-1-1 queue-1-2
> {noformat}
> Assume cluster resource is 100 and all queues have same weights.
> # queue-1-1 and queue-2 has apps. Each get 50 usage and 50 fairshare.
> # When queue-1-2 is active, supposedly it will preempt 25 from queue-1-1, but
> this doesn't happen because preemption happens top-down, queue-2 could be the
> preemption candidate as long as queue-2 is less needy than queue-1, and
> queue-2 doesn't exceed the fair share which means preemption won't happen.
> We need to filter out queue-2 since it isn't a valid candidate.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6141) ppc64le on Linux doesn't trigger __linux get_executable codepath
[
https://issues.apache.org/jira/browse/YARN-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860678#comment-15860678
]
Sonia Garudi commented on YARN-6141:
On upgrading the version of cmake to >3.1 the build is successful. For cmake
version < 3.1, the C standard is being set to c99 in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt:
if (CMAKE_VERSION VERSION_LESS "3.1")
# subset of CMAKE__COMPILER_ID
#
https://cmake.org/cmake/help/v3.0/variable/CMAKE_LANG_COMPILER_ID.html
if (CMAKE_C_COMPILER_ID STREQUAL "GNU" OR
CMAKE_C_COMPILER_ID STREQUAL "Clang" OR
CMAKE_C_COMPILER_ID STREQUAL "AppleClang")
set (CMAKE_C_FLAGS "-std=c99 -Wall -pedantic-errors
${CMAKE_C_FLAGS}")
elseif (CMAKE_C_COMPILER_ID STREQUAL "Intel")
set (CMAKE_C_FLAGS "-std=c99 -Wall ${CMAKE_C_FLAGS}")
elseif (CMAKE_C_COMPILER_ID STREQUAL "SunPro")
set (CMAKE_C_FLAGS "-xc99 ${CMAKE_C_FLAGS}")
endif ()
else ()
set (CMAKE_C_STANDARD 99)
endif ()
> ppc64le on Linux doesn't trigger __linux get_executable codepath
>
>
> Key: YARN-6141
> URL: https://issues.apache.org/jira/browse/YARN-6141
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha3
> Environment: $ uname -a
> Linux f8eef0f055cf 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Sonia Garudi
> Labels: ppc64le
> Attachments: YARN-6141.diff
>
>
> On ppc64le architecture, the build fails in the 'Hadoop YARN NodeManager'
> project with the below error:
> Cannot safely determine executable path with a relative HADOOP_CONF_DIR on
> this operating system.
> [WARNING] #error Cannot safely determine executable path with a relative
> HADOOP_CONF_DIR on this operating system.
> [WARNING] ^
> [WARNING] make[2]: ***
> [CMakeFiles/container.dir/main/native/container-executor/impl/get_executable.c.o]
> Error 1
> [WARNING] make[2]: *** Waiting for unfinished jobs
> [WARNING] make[1]: *** [CMakeFiles/container.dir/all] Error 2
> [WARNING] make: *** [all] Error 2
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> Cmake version used :
> $ /usr/bin/cmake --version
> cmake version 2.8.12.2
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6156) AM blacklistdisableThreshold consider node partition count
[ https://issues.apache.org/jira/browse/YARN-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-6156: --- Issue Type: Sub-task (was: Bug) Parent: YARN-4576 > AM blacklistdisableThreshold consider node partition count > --- > > Key: YARN-6156 > URL: https://issues.apache.org/jira/browse/YARN-6156 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6061) Add a customized uncaughtexceptionhandler for critical threads in RM
[
https://issues.apache.org/jira/browse/YARN-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860658#comment-15860658
]
ASF GitHub Bot commented on YARN-6061:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/182#discussion_r100468005
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMCriticalThreadUncaughtExceptionHandler.java
---
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager;
+
+import java.lang.Thread.UncaughtExceptionHandler;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.classification.InterfaceAudience.Public;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+import org.apache.hadoop.yarn.conf.HAUtil;
+
+/**
+ * This class either shuts down {@link ResourceManager} or transitions the
+ * {@link ResourceManager} to standby state if a critical thread throws an
+ * uncaught exception. It is intended to be installed by calling
+ * {@code setUncaughtExceptionHandler(Thread.UncaughtExceptionHandler)}
+ * in the thread entry point or after creation of threads.
+ */
+@Public
--- End diff --
We don't need to expose this outside YARN at all. This should be @Private.
Let us remove @Evolving altogether.
> Add a customized uncaughtexceptionhandler for critical threads in RM
>
>
> Key: YARN-6061
> URL: https://issues.apache.org/jira/browse/YARN-6061
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6061.001.patch, YARN-6061.002.patch,
> YARN-6061.003.patch, YARN-6061.004.patch, YARN-6061.005.patch,
> YARN-6061.006.patch, YARN-6061.007.patch, YARN-6061.008.patch
>
>
> There are several threads in fair scheduler. The thread will quit when there
> is a runtime exception inside it. We should bring down the RM when that
> happens. Otherwise, there may be some weird behavior in RM.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3933) Race condition when calling AbstractYarnScheduler.completedContainer.
[ https://issues.apache.org/jira/browse/YARN-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860593#comment-15860593 ] Miklos Szegedi commented on YARN-3933: -- I could not repro the unit test issues above. > Race condition when calling AbstractYarnScheduler.completedContainer. > - > > Key: YARN-3933 > URL: https://issues.apache.org/jira/browse/YARN-3933 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Lavkesh Lahngir >Assignee: Shiwei Guo > Labels: oct16-medium > Attachments: YARN-3933.001.patch, YARN-3933.002.patch, > YARN-3933.003.patch, YARN-3933.004.patch, YARN-3933.005.patch, > YARN-3933.006.patch > > > In our cluster we are seeing available memory and cores being negative. > Initial inspection: > Scenario no. 1: > In capacity scheduler the method allocateContainersToNode() checks if > there are excess reservation of containers for an application, and they are > no longer needed then it calls queue.completedContainer() which causes > resources being negative. And they were never assigned in the first place. > I am still looking through the code. Can somebody suggest how to simulate > excess containers assignments ? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6169) container-executor message on empty configuration file can be improved
[
https://issues.apache.org/jira/browse/YARN-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860590#comment-15860590
]
Miklos Szegedi commented on YARN-6169:
--
It would also be nice to print some additional information in the following
case:
{code}
if (user_info->pw_uid < min_uid && !is_whitelisted(user)) {
fprintf(LOGFILE, "Requested user %s is not whitelisted and has id %d,"
"which is below the minimum allowed %d\n", user, user_info->pw_uid,
min_uid);
fprintf(LOGFILE, "The allowed.system.users configuration entry can be used"
" to whitelist a user");
fflush(LOGFILE);
free(user_info);
return NULL;
}
{code}
> container-executor message on empty configuration file can be improved
> --
>
> Key: YARN-6169
> URL: https://issues.apache.org/jira/browse/YARN-6169
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Miklos Szegedi
>Priority: Trivial
>
> If the configuration file is empty, we get the following error message:
> {{Invalid configuration provided in /root/etc/hadoop/container-executor.cfg}}
> This is does not provide enough details to figure out what is the issue at
> the first glance. We should use something like 'Empty configuration file
> provided...'
> {code}
> if (cfg->size == 0) {
> fprintf(ERRORFILE, "Invalid configuration provided in %s\n", file_name);
> exit(INVALID_CONFIG_FILE);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3933) Race condition when calling AbstractYarnScheduler.completedContainer.
[
https://issues.apache.org/jira/browse/YARN-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860552#comment-15860552
]
Hadoop QA commented on YARN-3933:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 24s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService
|
| | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-3933 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12851961/YARN-3933.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 95cc0dc6953e 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 08f9397 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14874/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14874/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14874/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT
[jira] [Updated] (YARN-6164) Expose maximum-am-resource-percent to Hadoop clients
[ https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benson Qiu updated YARN-6164: - Description: To my knowledge, there is no way to programmatically obtain `yarn.scheduler.capacity.maximum-am-resource-percent`. I've tried looking at [YarnClient|https://hadoop.apache.org/docs/r2.7.2/api/index.html?org/apache/hadoop/yarn/client/api/YarnClient.html], [ResourceManager REST APIs|https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html], and [YarnCommands|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html] (via bin/yarn) was: To my knowledge, there is no way to programmatically obtain `yarn.scheduler.capacity.maximum-applications`. I've tried looking at [YarnClient|https://hadoop.apache.org/docs/r2.7.2/api/index.html?org/apache/hadoop/yarn/client/api/YarnClient.html], [ResourceManager REST APIs|https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html], and [YarnCommands|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html] (via bin/yarn) > Expose maximum-am-resource-percent to Hadoop clients > > > Key: YARN-6164 > URL: https://issues.apache.org/jira/browse/YARN-6164 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Benson Qiu > Attachments: YARN-6164.001.patch > > > To my knowledge, there is no way to programmatically obtain > `yarn.scheduler.capacity.maximum-am-resource-percent`. > I've tried looking at > [YarnClient|https://hadoop.apache.org/docs/r2.7.2/api/index.html?org/apache/hadoop/yarn/client/api/YarnClient.html], > [ResourceManager REST > APIs|https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html], > and > [YarnCommands|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html] > (via bin/yarn) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6164) Expose maximum-am-resource-percent to Hadoop clients
[ https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benson Qiu updated YARN-6164: - Target Version/s: 3.0.0-alpha3 > Expose maximum-am-resource-percent to Hadoop clients > > > Key: YARN-6164 > URL: https://issues.apache.org/jira/browse/YARN-6164 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Benson Qiu > Attachments: YARN-6164.001.patch > > > To my knowledge, there is no way to programmatically obtain > `yarn.scheduler.capacity.maximum-applications`. > I've tried looking at > [YarnClient|https://hadoop.apache.org/docs/r2.7.2/api/index.html?org/apache/hadoop/yarn/client/api/YarnClient.html], > [ResourceManager REST > APIs|https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html], > and > [YarnCommands|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html] > (via bin/yarn) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6164) Expose maximum-am-resource-percent to Hadoop clients
[ https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benson Qiu updated YARN-6164: - Attachment: YARN-6164.001.patch > Expose maximum-am-resource-percent to Hadoop clients > > > Key: YARN-6164 > URL: https://issues.apache.org/jira/browse/YARN-6164 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Benson Qiu > Attachments: YARN-6164.001.patch > > > To my knowledge, there is no way to programmatically obtain > `yarn.scheduler.capacity.maximum-applications`. > I've tried looking at > [YarnClient|https://hadoop.apache.org/docs/r2.7.2/api/index.html?org/apache/hadoop/yarn/client/api/YarnClient.html], > [ResourceManager REST > APIs|https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html], > and > [YarnCommands|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html] > (via bin/yarn) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860481#comment-15860481
]
Hadoop QA commented on YARN-6166:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
46s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6166 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12851955/YARN-6166.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 41e37dfb520b 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 08f9397 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14873/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14873/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: Grant W
>Priority: Trivial
> Labels: patch
>
[jira] [Created] (YARN-6169) container-executor message on empty configuration file can be improved
Miklos Szegedi created YARN-6169:
Summary: container-executor message on empty configuration file
can be improved
Key: YARN-6169
URL: https://issues.apache.org/jira/browse/YARN-6169
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Miklos Szegedi
Priority: Trivial
If the configuration file is empty, we get the following error message:
{{Invalid configuration provided in /root/etc/hadoop/container-executor.cfg}}
This is does not provide enough details to figure out what is the issue at the
first glance. We should use something like 'Empty configuration file
provided...'
{code}
if (cfg->size == 0) {
fprintf(ERRORFILE, "Invalid configuration provided in %s\n", file_name);
exit(INVALID_CONFIG_FILE);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6144) FairScheduler: preempted resources can become negative
[
https://issues.apache.org/jira/browse/YARN-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860445#comment-15860445
]
Miklos Szegedi commented on YARN-6144:
--
Thank you, [~kasha].
> FairScheduler: preempted resources can become negative
> --
>
> Key: YARN-6144
> URL: https://issues.apache.org/jira/browse/YARN-6144
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: Screen Shot 2017-02-02 at 12.49.14 PM.png,
> YARN-6144.000.patch, YARN-6144.001.patch, YARN-6144.002.patch,
> YARN-6144.003.patch
>
>
> {{preemptContainers()}} calls {{trackContainerForPreemption()}} to collect
> the list of containers and resources that were preempted for an application.
> Later the list is reduced when {{containerCompleted()}} calls
> {{untrackContainerForPreemption()}}. The bug is that the resource variable
> {{preemptedResources}} is subtracted, not just when the container was
> preempted but also when it has completed successfully. This causes that we
> return an incorrect value in {{getResourceUsage()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6144) FairScheduler: preempted resources can become negative
[
https://issues.apache.org/jira/browse/YARN-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860443#comment-15860443
]
Hudson commented on YARN-6144:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11229 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11229/])
YARN-6144. FairScheduler: preempted resources can become negative. (kasha: rev
08f93978f3ec724b24a93d7ef538f158da75802f)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
> FairScheduler: preempted resources can become negative
> --
>
> Key: YARN-6144
> URL: https://issues.apache.org/jira/browse/YARN-6144
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: Screen Shot 2017-02-02 at 12.49.14 PM.png,
> YARN-6144.000.patch, YARN-6144.001.patch, YARN-6144.002.patch,
> YARN-6144.003.patch
>
>
> {{preemptContainers()}} calls {{trackContainerForPreemption()}} to collect
> the list of containers and resources that were preempted for an application.
> Later the list is reduced when {{containerCompleted()}} calls
> {{untrackContainerForPreemption()}}. The bug is that the resource variable
> {{preemptedResources}} is subtracted, not just when the container was
> preempted but also when it has completed successfully. This causes that we
> return an incorrect value in {{getResourceUsage()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6113) re-direct NM Web Service to get container logs for finished applications
[ https://issues.apache.org/jira/browse/YARN-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860436#comment-15860436 ] Xuan Gong commented on YARN-6113: - [~djp] The test case failure is not related > re-direct NM Web Service to get container logs for finished applications > > > Key: YARN-6113 > URL: https://issues.apache.org/jira/browse/YARN-6113 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-6113.branch-2.v1.patch, > YARN-6113.branch-2.v2.patch, YARN-6113.branch-2.v3.patch, > YARN-6113.branch-2.v4.patch, YARN-6113.trunk.v2.patch, > YARN-6113.trunk.v3.patch, YARN-6113.trunk.v4.patch > > > In NM web ui, when we try to get container logs for finished application, it > would redirect to the log server based on the configuration: > yarn.log.server.url. We should do the similar thing for NM WebService -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3933) Race condition when calling AbstractYarnScheduler.completedContainer.
[ https://issues.apache.org/jira/browse/YARN-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-3933: - Attachment: YARN-3933.006.patch [~guoshiwei], thank you for looking into this. I ran into the same issue in YARN-6158 and I had a patch there. I attach the rebased version here for your consideration. It is very similar to the current patch, it just addresses the test issue. I am also okay with 004.patch. It would be nice to get either of the fixes checked in. What do you think? > Race condition when calling AbstractYarnScheduler.completedContainer. > - > > Key: YARN-3933 > URL: https://issues.apache.org/jira/browse/YARN-3933 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Lavkesh Lahngir >Assignee: Shiwei Guo > Labels: oct16-medium > Attachments: YARN-3933.001.patch, YARN-3933.002.patch, > YARN-3933.003.patch, YARN-3933.004.patch, YARN-3933.005.patch, > YARN-3933.006.patch > > > In our cluster we are seeing available memory and cores being negative. > Initial inspection: > Scenario no. 1: > In capacity scheduler the method allocateContainersToNode() checks if > there are excess reservation of containers for an application, and they are > no longer needed then it calls queue.completedContainer() which causes > resources being negative. And they were never assigned in the first place. > I am still looking through the code. Can somebody suggest how to simulate > excess containers assignments ? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5501) Container Pooling in YARN
[ https://issues.apache.org/jira/browse/YARN-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860407#comment-15860407 ] Hitesh Sharma commented on YARN-5501: - Thanks for the continued feedback, [~jlowe]. bq. Thanks for the detailed answers. I highly recommend these get captured in a new revision of the design document along with diagrams to help others come up to speed and join the discussion. Otherwise we have a wall of text in this JIRA that is essential reading for anyone understanding what is really being proposed. Sure, I will do that. bq. My thinking is the mere fact that the container finishes is indicative that it is ready for reuse. Maybe there's an explicit API they call at the end of the task or whatever, but the same has to be done for this existing design as well -- we need to know when the container is ready to be reused. The main difference I see in this approach is that there isn't an explicit 'pre-init' step where the users or admins need to premeditate what will be run. Instead the first run of the app framework is the same performance it is today, but subsequent runs are faster since it can reused those cached containers. Seems to me the most difficult part of this is coming up with an efficient container request protocol so YARN can know when it can reuse an old, cached container and when it cannot. The existing proposal works around this by requiring the containers to be setup beforehand as special resource types, but that won't work for a general container caching approach. Yes, once we have a way to know that the container is ready to be reused then we can issue a detach on it and add it to the container pool. We had a similar issue in our PoC where we needed to know that the container is pre-initialized or not (as the launched process can take some minutes to be fully ready). As you also said there is no protocol between YARN NM and the container to know what's going on, so we ended up looking for a trigger file in the pre-init container working directory to detect that pre-initialization is done, and the container can be inducted into the pool after that. We can use something similar to this for e.g. containers can create a trigger file and launcher looks for that. If found it detaches the container and inducts it into the pool. bq. They certainly are enforced, but how does the app know about the new constraints so they can either avoid getting shot or take advantage of the new space? Simply updating the cgroup is not going to be sufficient. Either the process will OOM because it slams into the new lower limit (potentially instantly if it is already bigger than the new limit) or it will be completely oblivious that it now has acres of memory that it can use. If it tried to use it before it would fail, so how does it know it grew? For example, the JVM can't magically do this unless the app is doing some sort of explicit off-heap memory management via direct buffers, etc. and is told about its memory limit. Simply updating the cgroup setting doesn't seem to be a sufficient communication channel here, so I'm curious how that's all you need to do for your scenario. It is ok for our scenario because the process we pre-initialize don't do any work and simply initialize themselves. These processes also happen to be not JVM in general (and where we do use JVM I don't think we specify resource limits when starting the processes). When the actual AM comes around and issues work then the processes started a priori require resources and that time we adjust the cgroup or job object. Strictly speaking we aren't using resource resizing as it is in YARN but have our own mechanisms to update the resource constraints. > Container Pooling in YARN > - > > Key: YARN-5501 > URL: https://issues.apache.org/jira/browse/YARN-5501 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun Suresh >Assignee: Hitesh Sharma > Attachments: Container Pooling - one pager.pdf > > > This JIRA proposes a method for reducing the container launch latency in > YARN. It introduces a notion of pooling *Unattached Pre-Initialized > Containers*. > Proposal in brief: > * Have a *Pre-Initialized Container Factory* service within the NM to create > these unattached containers. > * The NM would then advertise these containers as special resource types > (this should be possible via YARN-3926). > * When a start container request is received by the node manager for > launching a container requesting this specific type of resource, it will take > one of these unattached pre-initialized containers from the pool, and use it > to service the container request. > * Once the request is complete, the pre-initialized container would be > released and ready to serve another
[jira] [Updated] (YARN-6144) FairScheduler: preempted resources can become negative
[
https://issues.apache.org/jira/browse/YARN-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6144:
---
Fix Version/s: 3.0.0-alpha2
> FairScheduler: preempted resources can become negative
> --
>
> Key: YARN-6144
> URL: https://issues.apache.org/jira/browse/YARN-6144
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: Screen Shot 2017-02-02 at 12.49.14 PM.png,
> YARN-6144.000.patch, YARN-6144.001.patch, YARN-6144.002.patch,
> YARN-6144.003.patch
>
>
> {{preemptContainers()}} calls {{trackContainerForPreemption()}} to collect
> the list of containers and resources that were preempted for an application.
> Later the list is reduced when {{containerCompleted()}} calls
> {{untrackContainerForPreemption()}}. The bug is that the resource variable
> {{preemptedResources}} is subtracted, not just when the container was
> preempted but also when it has completed successfully. This causes that we
> return an incorrect value in {{getResourceUsage()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6112) UpdateCallDuration is calculated only when debug logging is enabled
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6112:
---
Fix Version/s: 3.0.0-alpha3
> UpdateCallDuration is calculated only when debug logging is enabled
> ---
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6112.001.patch, YARN-6112.002.patch,
> YARN-6112.003.patch, YARN-6112.branch-2.001.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6144) FairScheduler: preempted resources can become negative
[
https://issues.apache.org/jira/browse/YARN-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6144:
---
Fix Version/s: (was: 3.0.0-alpha2)
3.0.0-alpha3
> FairScheduler: preempted resources can become negative
> --
>
> Key: YARN-6144
> URL: https://issues.apache.org/jira/browse/YARN-6144
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: Screen Shot 2017-02-02 at 12.49.14 PM.png,
> YARN-6144.000.patch, YARN-6144.001.patch, YARN-6144.002.patch,
> YARN-6144.003.patch
>
>
> {{preemptContainers()}} calls {{trackContainerForPreemption()}} to collect
> the list of containers and resources that were preempted for an application.
> Later the list is reduced when {{containerCompleted()}} calls
> {{untrackContainerForPreemption()}}. The bug is that the resource variable
> {{preemptedResources}} is subtracted, not just when the container was
> preempted but also when it has completed successfully. This causes that we
> return an incorrect value in {{getResourceUsage()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6164) Expose maximum-am-resource-percent to Hadoop clients
[ https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benson Qiu updated YARN-6164: - Description: To my knowledge, there is no way to programmatically obtain `yarn.scheduler.capacity.maximum-applications`. I've tried looking at [YarnClient|https://hadoop.apache.org/docs/r2.7.2/api/index.html?org/apache/hadoop/yarn/client/api/YarnClient.html], [ResourceManager REST APIs|https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html], and [YarnCommands|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html] (via bin/yarn) was: To my knowledge, there is no way to programmatically obtain `yarn.scheduler.capacity.maximum-applications`. I've tried looking at [YarnClient|https://hadoop.apache.org/docs/r2.7.2/api/index.html?org/apache/hadoop/yarn/client/api/YarnClient.html], [ResourceManager REST APIs|https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html], and [YarnCommands|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html] (via bin/yarn.sh) > Expose maximum-am-resource-percent to Hadoop clients > > > Key: YARN-6164 > URL: https://issues.apache.org/jira/browse/YARN-6164 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Benson Qiu > > To my knowledge, there is no way to programmatically obtain > `yarn.scheduler.capacity.maximum-applications`. > I've tried looking at > [YarnClient|https://hadoop.apache.org/docs/r2.7.2/api/index.html?org/apache/hadoop/yarn/client/api/YarnClient.html], > [ResourceManager REST > APIs|https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html], > and > [YarnCommands|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html] > (via bin/yarn) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6168) Restarted RM may not inform AM about all existing containers
Billie Rinaldi created YARN-6168: Summary: Restarted RM may not inform AM about all existing containers Key: YARN-6168 URL: https://issues.apache.org/jira/browse/YARN-6168 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi There appears to be a race condition when an RM is restarted. I had a situation where the RMs and AM were down, but NMs and app containers were still running. When I restarted the RM, the AM restarted, registered with the RM, and received its list of existing containers before the NMs had reported all of their containers to the RM. The AM was only told about some of the app's existing containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860358#comment-15860358
]
Daniel Templeton commented on YARN-6166:
LGTM. +1
Next, time, though, it's better to keep all the patches attached to the JIRA so
that the history is preserved. To avoid confusion, we usually name them
YARN-6166.001.patch, YARN-6166.002.patch, etc.
I'll commit it in a bit.
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: Grant W
>Priority: Trivial
> Labels: patch
> Attachments: YARN-6166.patch
>
>
> Logs like the following should be debug or else every legitimate stop causes
> unnecessary exception traces in the logs.
> {noformat}
> 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
> Interrupted while waiting for queue
> java.lang.InterruptedException
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:1961)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-3666) Federation Intercepting and propagating AM-RM communications
[ https://issues.apache.org/jira/browse/YARN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan reassigned YARN-3666: Assignee: Botong Huang (was: Kishore Chaliparambil) > Federation Intercepting and propagating AM-RM communications > > > Key: YARN-3666 > URL: https://issues.apache.org/jira/browse/YARN-3666 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Kishore Chaliparambil >Assignee: Botong Huang > > In order, to support transparent "spanning" of jobs across sub-clusters, all > AM-RM communications are proxied (via YARN-2884). > This JIRA tracks federation-specific mechanisms that decide how to > "split/broadcast" requests to the RMs and "merge" answers to > the AM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5531) UnmanagedAM pool manager for federating application across clusters
[ https://issues.apache.org/jira/browse/YARN-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan reassigned YARN-5531: Assignee: Botong Huang (was: Sarvesh Sakalanaga) > UnmanagedAM pool manager for federating application across clusters > --- > > Key: YARN-5531 > URL: https://issues.apache.org/jira/browse/YARN-5531 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Botong Huang > > One of the main tenets the YARN Federation is to *transparently* scale > applications across multiple clusters. This is achieved by running UAMs on > behalf of the application on other clusters. This JIRA tracks the addition of > a UnmanagedAM pool manager for federating application across clusters which > will be used the FederationInterceptor (YARN-3666) which is part of the > AMRMProxy pipeline introduced in YARN-2884. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6164) Expose maximum-am-resource-percent to Hadoop clients
[ https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860357#comment-15860357 ] Benson Qiu commented on YARN-6164: -- I built the latest version of Hadoop and it appears that [maxAMLimitPercentage|https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueCapacitiesInfo.java#L52] has been added to the Cluster Scheduler API (/ws/v1/cluster/scheduler). I still don't believe this config is exposed through YarnClient or YarnCLI though. > Expose maximum-am-resource-percent to Hadoop clients > > > Key: YARN-6164 > URL: https://issues.apache.org/jira/browse/YARN-6164 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Benson Qiu > > To my knowledge, there is no way to programmatically obtain > `yarn.scheduler.capacity.maximum-applications`. > I've tried looking at > [YarnClient|https://hadoop.apache.org/docs/r2.7.2/api/index.html?org/apache/hadoop/yarn/client/api/YarnClient.html], > [ResourceManager REST > APIs|https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html], > and > [YarnCommands|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html] > (via bin/yarn.sh) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860349#comment-15860349
]
Grant W commented on YARN-6166:
---
I've taken another crack at the patch. Hopefully I don't continue to embarrass
myself.
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: Grant W
>Priority: Trivial
> Labels: patch
> Attachments: YARN-6166.patch
>
>
> Logs like the following should be debug or else every legitimate stop causes
> unnecessary exception traces in the logs.
> {noformat}
> 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
> Interrupted while waiting for queue
> java.lang.InterruptedException
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:1961)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6167) RM option to delegate NM loss container action to AM
Billie Rinaldi created YARN-6167: Summary: RM option to delegate NM loss container action to AM Key: YARN-6167 URL: https://issues.apache.org/jira/browse/YARN-6167 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Reporter: Billie Rinaldi Fix For: yarn-native-services Currently, if the RM times out an NM, the scheduler will kill all containers that were running on the NM. For some applications, in the event of a temporary NM outage, it might be better to delegate to the AM the decision whether to kill the containers and request new containers from the RM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant W updated YARN-6166:
--
Attachment: YARN-6166.patch
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: Grant W
>Priority: Trivial
> Labels: patch
> Attachments: YARN-6166.patch
>
>
> Logs like the following should be debug or else every legitimate stop causes
> unnecessary exception traces in the logs.
> {noformat}
> 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
> Interrupted while waiting for queue
> java.lang.InterruptedException
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:1961)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant W updated YARN-6166:
--
Attachment: (was: YARN-6166.patch)
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: Grant W
>Priority: Trivial
> Labels: patch
> Attachments: YARN-6166.patch
>
>
> Logs like the following should be debug or else every legitimate stop causes
> unnecessary exception traces in the logs.
> {noformat}
> 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
> Interrupted while waiting for queue
> java.lang.InterruptedException
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:1961)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned YARN-6166:
--
Assignee: Grant W (was: haosdent)
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: Grant W
>Priority: Trivial
> Labels: patch
> Attachments: YARN-6166.patch
>
>
> Logs like the following should be debug or else every legitimate stop causes
> unnecessary exception traces in the logs.
> {noformat}
> 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
> Interrupted while waiting for queue
> java.lang.InterruptedException
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:1961)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860342#comment-15860342
]
Daniel Templeton commented on YARN-6166:
I assigned the JIRA to you, [~genericuser]. {{interrupt()}} isn't a static
method, so I doubt that patch will compile. You'll have to get the current
thread to call {{interrupt()}} on it. See
https://www.ibm.com/developerworks/library/j-jtp05236/.
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: Grant W
>Priority: Trivial
> Labels: patch
> Attachments: YARN-6166.patch
>
>
> Logs like the following should be debug or else every legitimate stop causes
> unnecessary exception traces in the logs.
> {noformat}
> 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
> Interrupted while waiting for queue
> java.lang.InterruptedException
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:1961)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6056) Yarn NM using LCE shows a failure when trying to delete a non-existing dir
[ https://issues.apache.org/jira/browse/YARN-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860337#comment-15860337 ] Jonathan Hung commented on YARN-6056: - Thanks [~varun_saxena]! Any update on this? > Yarn NM using LCE shows a failure when trying to delete a non-existing dir > -- > > Key: YARN-6056 > URL: https://issues.apache.org/jira/browse/YARN-6056 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.6.4 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg > Attachments: YARN-6056-branch-2.6.01.patch, > YARN-6056-branch-2.6.1.patch > > > As part of YARN-2902 the clean up of the local directories was changed to > ignore non existing directories and proceed with others in the list. This > part of the code change was not backported into branch-2.6, backporting just > that part now. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860332#comment-15860332
]
Grant W commented on YARN-6166:
---
[~haosd...@gmail.com] got pulled in from the previous issue I cloned. I don't
seem to have any way to remove him from assignee.
I attached a new patch file with the interrupt call.
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: haosdent
>Priority: Trivial
> Labels: patch
> Attachments: YARN-6166.patch
>
>
> Logs like the following should be debug or else every legitimate stop causes
> unnecessary exception traces in the logs.
> {noformat}
> 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
> Interrupted while waiting for queue
> java.lang.InterruptedException
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:1961)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant W updated YARN-6166:
--
Attachment: YARN-6166.patch
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: haosdent
>Priority: Trivial
> Labels: patch
> Attachments: YARN-6166.patch
>
>
> Logs like the following should be debug or else every legitimate stop causes
> unnecessary exception traces in the logs.
> {noformat}
> 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
> Interrupted while waiting for queue
> java.lang.InterruptedException
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:1961)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant W updated YARN-6166:
--
Attachment: (was: YARN-6166.patch)
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: haosdent
>Priority: Trivial
> Labels: patch
>
> Logs like the following should be debug or else every legitimate stop causes
> unnecessary exception traces in the logs.
> {noformat}
> 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
> Interrupted while waiting for queue
> java.lang.InterruptedException
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:1961)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6158) FairScheduler: app usage can go to negative
[
https://issues.apache.org/jira/browse/YARN-6158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860301#comment-15860301
]
Miklos Szegedi edited comment on YARN-6158 at 2/9/17 10:18 PM:
---
I just noticed, I think it is the same issue as YARN-3933. This patch copies,
what CapacityScheduler does, so it helps any refactoring in the future to merge
some codebase.
was (Author: miklos.szeg...@cloudera.com):
I just noticed, think it is the same issue as YARN-3933. This patch copies,
what CapacityScheduler does, so it helps any refactoring in the future to merge
some codebase.
> FairScheduler: app usage can go to negative
> ---
>
> Key: YARN-6158
> URL: https://issues.apache.org/jira/browse/YARN-6158
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6158.000.patch
>
>
> FiCaSchedulerApp.containerCompleted checks, if the container being completed
> is in the active list:
> {code}
> // Remove from the list of containers
> if (null == liveContainers.remove(containerId)) {
> return false;
> }
> {code}
> Fair scheduler should do the same, otherwise multiple different container
> close events leave the application with negative resource usage in
> {{queue.getMetrics().releaseResources}} and {{attemptResourceUsage.decUsed}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6144) FairScheduler: preempted resources can become negative
[
https://issues.apache.org/jira/browse/YARN-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860299#comment-15860299
]
Hadoop QA commented on YARN-6144:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 201 unchanged - 1 fixed = 202 total (was 202)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 11s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6144 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12851944/YARN-6144.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ca6a75d720ee 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5fb723b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14872/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14872/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14872/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
[jira] [Commented] (YARN-6158) FairScheduler: app usage can go to negative
[
https://issues.apache.org/jira/browse/YARN-6158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860301#comment-15860301
]
Miklos Szegedi commented on YARN-6158:
--
I just noticed, think it is the same issue as YARN-3933. This patch copies,
what CapacityScheduler does, so it helps any refactoring in the future to merge
some codebase.
> FairScheduler: app usage can go to negative
> ---
>
> Key: YARN-6158
> URL: https://issues.apache.org/jira/browse/YARN-6158
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6158.000.patch
>
>
> FiCaSchedulerApp.containerCompleted checks, if the container being completed
> is in the active list:
> {code}
> // Remove from the list of containers
> if (null == liveContainers.remove(containerId)) {
> return false;
> }
> {code}
> Fair scheduler should do the same, otherwise multiple different container
> close events leave the application with negative resource usage in
> {{queue.getMetrics().releaseResources}} and {{attemptResourceUsage.decUsed}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860277#comment-15860277
]
Daniel Templeton commented on YARN-6166:
Thanks for the patch, [~haosd...@gmail.com]. Could you also add a call to
{{Thread.interrupt()}} before the {{continue}}?
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: haosdent
>Priority: Trivial
> Labels: patch
> Attachments: YARN-6166.patch
>
>
> Logs like the following should be debug or else every legitimate stop causes
> unnecessary exception traces in the logs.
> {noformat}
> 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
> Interrupted while waiting for queue
> java.lang.InterruptedException
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:1961)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860245#comment-15860245
]
Hadoop QA commented on YARN-6166:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 25s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMProxy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6166 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12851942/YARN-6166.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ede92cdad065 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5fb723b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14871/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14871/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14871/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL:
[jira] [Commented] (YARN-6112) UpdateCallDuration is calculated only when debug logging is enabled
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860241#comment-15860241
]
Hudson commented on YARN-6112:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11228 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11228/])
YARN-6112. UpdateCallDuration is calculated only when debug logging is (kasha:
rev 9b85053583a3560f93062b656061d11b1b9c664f)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSOpDurations.java
> UpdateCallDuration is calculated only when debug logging is enabled
> ---
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0
>
> Attachments: YARN-6112.001.patch, YARN-6112.002.patch,
> YARN-6112.003.patch, YARN-6112.branch-2.001.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[
https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6166:
---
Description:
Logs like the following should be debug or else every legitimate stop causes
unnecessary exception traces in the logs.
{noformat}
2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
Interrupted while waiting for queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
java:1961)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
{noformat}
was:
Logs like the following should be debug or else every legitimate stop causes
unnecessary exception traces in the logs.
2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
Interrupted while waiting for queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
java:1961)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
> --
>
> Key: YARN-6166
> URL: https://issues.apache.org/jira/browse/YARN-6166
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Grant W
>Assignee: haosdent
>Priority: Trivial
> Labels: patch
> Attachments: YARN-6166.patch
>
>
> Logs like the following should be debug or else every legitimate stop causes
> unnecessary exception traces in the logs.
> {noformat}
> 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread]
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl:
> Interrupted while waiting for queue
> java.lang.InterruptedException
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:1961)
>at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
>at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6112) UpdateCallDuration is calculated only when debug logging is enabled
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860212#comment-15860212
]
Yufei Gu commented on YARN-6112:
Thanks [~kasha] for the review and commit.
> UpdateCallDuration is calculated only when debug logging is enabled
> ---
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0
>
> Attachments: YARN-6112.001.patch, YARN-6112.002.patch,
> YARN-6112.003.patch, YARN-6112.branch-2.001.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6112) UpdateCallDuration is calculated only when debug logging is enabled
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6112:
---
Summary: UpdateCallDuration is calculated only when debug logging is
enabled (was: fsOpDurations.addUpdateCallDuration() should be independent to
LOG level)
> UpdateCallDuration is calculated only when debug logging is enabled
> ---
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6112.001.patch, YARN-6112.002.patch,
> YARN-6112.003.patch, YARN-6112.branch-2.001.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6144) FairScheduler: preempted resources can become negative
[
https://issues.apache.org/jira/browse/YARN-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated YARN-6144:
-
Attachment: YARN-6144.003.patch
[~kasha], that is a good point. Updating patch.
> FairScheduler: preempted resources can become negative
> --
>
> Key: YARN-6144
> URL: https://issues.apache.org/jira/browse/YARN-6144
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Blocker
> Attachments: Screen Shot 2017-02-02 at 12.49.14 PM.png,
> YARN-6144.000.patch, YARN-6144.001.patch, YARN-6144.002.patch,
> YARN-6144.003.patch
>
>
> {{preemptContainers()}} calls {{trackContainerForPreemption()}} to collect
> the list of containers and resources that were preempted for an application.
> Later the list is reduced when {{containerCompleted()}} calls
> {{untrackContainerForPreemption()}}. The bug is that the resource variable
> {{preemptedResources}} is subtracted, not just when the container was
> preempted but also when it has completed successfully. This causes that we
> return an incorrect value in {{getResourceUsage()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[ https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant W updated YARN-6166: -- Attachment: YARN-6166.patch > Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run > -- > > Key: YARN-6166 > URL: https://issues.apache.org/jira/browse/YARN-6166 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Grant W >Assignee: haosdent >Priority: Trivial > Labels: patch > Attachments: YARN-6166.patch > > > Logs like the following should be debug or else every legitimate stop causes > unnecessary exception traces in the logs. > 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread] > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: > Interrupted while waiting for queue > java.lang.InterruptedException >at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer. > java:1961) >at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996) >at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) >at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6162) Cluster Writeable APIs in RM Rest APIs page refers to APIs as alpha but they are already in use with Spark
[
https://issues.apache.org/jira/browse/YARN-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860177#comment-15860177
]
Grant Sohn commented on YARN-6162:
--
I'd say the APIs are stable enough that they shouldn't be labelled alpha.
Searching on this API reveals lots of links
from 2015 to now and some libraries/ports so it isn't really screaming bleeding
edge anymore.
> Cluster Writeable APIs in RM Rest APIs page refers to APIs as alpha but they
> are already in use with Spark
> --
>
> Key: YARN-6162
> URL: https://issues.apache.org/jira/browse/YARN-6162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation, site
>Reporter: Grant Sohn
>Assignee: Grant Sohn
>Priority: Trivial
>
> Excerpt with documentation that should be removed.
> {quote}
> Cluster Writeable APIs
> The setions below refer to APIs which allow to create and modify
> applications. -These APIs are currently in alpha and may change in the
> future.-
> Cluster New Application API
> With the New Application API, you can obtain an application-id which can then
> be used as part of the Cluster Submit Applications API to submit
> applications. The response also includes the maximum resource capabilities
> available on the cluster.
> -This feature is currently in the alpha stage and may change in the future.-
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2
[ https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860164#comment-15860164 ] Sangjin Lee commented on YARN-4675: --- We should be good to go once you addressed Varun's comments. Should this go to our feature branch (YARN-5355) or to trunk directly? I would think our feature branch should be the target? If so, could you kindly regenerate the patch against the branch? > Reorganize TimelineClient and TimelineClientImpl into separate classes for > ATSv1.x and ATSv2 > > > Key: YARN-4675 > URL: https://issues.apache.org/jira/browse/YARN-4675 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, > YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, > YARN-4675.v2.007.patch, YARN-4675-YARN-2928.v1.001.patch > > > We need to reorganize TimeClientImpl into TimeClientV1Impl , > TimeClientV2Impl and if required a base class, so that its clear which part > of the code belongs to which version and thus better maintainable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[ https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Whiteheart updated YARN-6166: --- Description: Logs like the following should be debug or else every legitimate stop causes unnecessary exception traces in the logs. 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread] org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: Interrupted while waiting for queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer. java:1961) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275) was: Logs like the following should be debug or else every legitimate stop causes unnecessary exception traces in the logs. 464 2013-08-03 20:01:34,459 INFO [AMRM Heartbeater thread] org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: Heartbeater interrupted 465 java.lang.InterruptedException: sleep interrupted 466 at java.lang.Thread.sleep(Native Method) 467 at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:249) 468 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread] org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: Interrupted while waiting for queue 469 java.lang.InterruptedException 470 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer. java:1961) 471 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996) 472 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) 473 at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275) > Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run > -- > > Key: YARN-6166 > URL: https://issues.apache.org/jira/browse/YARN-6166 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Grant Whiteheart >Assignee: haosdent >Priority: Trivial > Labels: newbie > > Logs like the following should be debug or else every legitimate stop causes > unnecessary exception traces in the logs. > 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread] > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: > Interrupted while waiting for queue > java.lang.InterruptedException >at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer. > java:1961) >at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996) >at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) >at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[ https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Whiteheart updated YARN-6166: --- Fix Version/s: (was: 2.3.0) > Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run > -- > > Key: YARN-6166 > URL: https://issues.apache.org/jira/browse/YARN-6166 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Grant Whiteheart >Assignee: haosdent >Priority: Trivial > Labels: newbie > > Logs like the following should be debug or else every legitimate stop causes > unnecessary exception traces in the logs. > 464 2013-08-03 20:01:34,459 INFO [AMRM Heartbeater thread] > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: > Heartbeater interrupted > 465 java.lang.InterruptedException: sleep interrupted > 466 at java.lang.Thread.sleep(Native Method) > 467 at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:249) > 468 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread] > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: > Interrupted while waiting for queue > 469 java.lang.InterruptedException > 470 at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer. > java:1961) > 471 at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996) > 472 at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > 473 at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[ https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Whiteheart updated YARN-6166: --- Affects Version/s: (was: 2.3.0) (was: 2.2.0) 2.7.3 > Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run > -- > > Key: YARN-6166 > URL: https://issues.apache.org/jira/browse/YARN-6166 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Grant Whiteheart >Assignee: haosdent >Priority: Trivial > Labels: newbie > Fix For: 2.3.0 > > > Logs like the following should be debug or else every legitimate stop causes > unnecessary exception traces in the logs. > 464 2013-08-03 20:01:34,459 INFO [AMRM Heartbeater thread] > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: > Heartbeater interrupted > 465 java.lang.InterruptedException: sleep interrupted > 466 at java.lang.Thread.sleep(Native Method) > 467 at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:249) > 468 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread] > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: > Interrupted while waiting for queue > 469 java.lang.InterruptedException > 470 at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer. > java:1961) > 471 at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996) > 472 at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > 473 at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[ https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Whiteheart updated YARN-6166: --- Target Version/s: 2.7.4 (was: 2.3.0) > Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run > -- > > Key: YARN-6166 > URL: https://issues.apache.org/jira/browse/YARN-6166 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Grant Whiteheart >Assignee: haosdent >Priority: Trivial > Labels: newbie > Fix For: 2.3.0 > > > Logs like the following should be debug or else every legitimate stop causes > unnecessary exception traces in the logs. > 464 2013-08-03 20:01:34,459 INFO [AMRM Heartbeater thread] > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: > Heartbeater interrupted > 465 java.lang.InterruptedException: sleep interrupted > 466 at java.lang.Thread.sleep(Native Method) > 467 at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:249) > 468 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread] > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: > Interrupted while waiting for queue > 469 java.lang.InterruptedException > 470 at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer. > java:1961) > 471 at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996) > 472 at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > 473 at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
[ https://issues.apache.org/jira/browse/YARN-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860138#comment-15860138 ] Grant Whiteheart commented on YARN-6166: YARN-1022 fixed unnecessary log messages from AMRMClientAsyncImpl$HeartbeatThread.run, but we still have unnecessary log messages from AMRMClientAsyncImpl$CallbackHandlerThread.run. 17/02/09 13:27:42 INFO impl.AMRMClientAsyncImpl: Interrupted while waiting for queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274) > Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run > -- > > Key: YARN-6166 > URL: https://issues.apache.org/jira/browse/YARN-6166 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0, 2.3.0 >Reporter: Grant Whiteheart >Assignee: haosdent >Priority: Trivial > Labels: newbie > Fix For: 2.3.0 > > > Logs like the following should be debug or else every legitimate stop causes > unnecessary exception traces in the logs. > 464 2013-08-03 20:01:34,459 INFO [AMRM Heartbeater thread] > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: > Heartbeater interrupted > 465 java.lang.InterruptedException: sleep interrupted > 466 at java.lang.Thread.sleep(Native Method) > 467 at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:249) > 468 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread] > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: > Interrupted while waiting for queue > 469 java.lang.InterruptedException > 470 at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer. > java:1961) > 471 at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996) > 472 at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > 473 at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860136#comment-15860136 ] Sangjin Lee commented on YARN-4985: --- Thanks for refreshing my memory on this Haibo. Yes, I recall that was the main issue. To tease out client and server correctly, we'd need a separate (common) module both client and server depends on. The only way we could avoid it is if all the fine-grained downstream dependencies can be isolated and moved into the hbase-server module. Do you know if it is feasible? If that is not feasible, maybe this refactoring would be more problematic than beneficial. What would we lose if we didn't do this refactoring? Also, if we didn't do this refactoring and kept thins as is (just timelineservice-hbase), do we still have the dependency issues in terms of installing the coprocessor (we need to follow all transitive dependencies between classes)? If so, that needs to be addressed no matter what. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6166) Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run
Grant Whiteheart created YARN-6166: -- Summary: Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run Key: YARN-6166 URL: https://issues.apache.org/jira/browse/YARN-6166 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0, 2.3.0 Reporter: Grant Whiteheart Assignee: haosdent Priority: Trivial Fix For: 2.3.0 Logs like the following should be debug or else every legitimate stop causes unnecessary exception traces in the logs. 464 2013-08-03 20:01:34,459 INFO [AMRM Heartbeater thread] org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: Heartbeater interrupted 465 java.lang.InterruptedException: sleep interrupted 466 at java.lang.Thread.sleep(Native Method) 467 at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:249) 468 2013-08-03 20:01:34,460 INFO [AMRM Callback Handler Thread] org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: Interrupted while waiting for queue 469 java.lang.InterruptedException 470 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer. java:1961) 471 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996) 472 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) 473 at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Improve /flows API for more flexible filters fromid, collapse, userid
[ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860111#comment-15860111 ] Sangjin Lee commented on YARN-6027: --- We discussed this offline during our status call. Just to record what we agreed, can we separate the aspect of the collapse filter from this JIRA and focus on fromId and userId? I don't think there is an agreement on the collapse part, and we can still make progress with the other two filters separately. Thanks! > Improve /flows API for more flexible filters fromid, collapse, userid > - > > Key: YARN-6027 > URL: https://issues.apache.org/jira/browse/YARN-6027 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6027-YARN-5355.0001.patch > > > In YARN-5585 , fromId is supported for retrieving entities. We need similar > filter for flows/flowRun apps and flow run and flow as well. > Along with supporting fromId, this JIRA should also discuss following points > * Should we throw an exception for entities/entity retrieval if duplicates > found? > * TimelieEntity : > ** Should equals method also check for idPrefix? > ** Does idPrefix is part of identifiers? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6162) Cluster Writeable APIs in RM Rest APIs page refers to APIs as alpha but they are already in use with Spark
[
https://issues.apache.org/jira/browse/YARN-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860079#comment-15860079
]
Daniel Templeton commented on YARN-6162:
Is the issue that the APIs are mislabeled or that Spark is using an alpha API?
> Cluster Writeable APIs in RM Rest APIs page refers to APIs as alpha but they
> are already in use with Spark
> --
>
> Key: YARN-6162
> URL: https://issues.apache.org/jira/browse/YARN-6162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation, site
>Reporter: Grant Sohn
>Assignee: Grant Sohn
>Priority: Trivial
>
> Excerpt with documentation that should be removed.
> {quote}
> Cluster Writeable APIs
> The setions below refer to APIs which allow to create and modify
> applications. -These APIs are currently in alpha and may change in the
> future.-
> Cluster New Application API
> With the New Application API, you can obtain an application-id which can then
> be used as part of the Cluster Submit Applications API to submit
> applications. The response also includes the maximum resource capabilities
> available on the cluster.
> -This feature is currently in the alpha stage and may change in the future.-
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6158) FairScheduler: app usage can go to negative
[
https://issues.apache.org/jira/browse/YARN-6158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860057#comment-15860057
]
Hadoop QA commented on YARN-6158:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 43s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6158 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12851915/YARN-6158.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a2c1e8fd98ff 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5fb723b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14870/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14870/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14870/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FairScheduler: app usage can go to negative
> ---
>
> Key: YARN-6158
>
[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860008#comment-15860008 ] Yufei Gu commented on YARN-4090: Thanks [~zsl2007]. I think you get it covered. Things might affect resource usage would be: # Submit an application # Allocate tasks # Move app from one queue to another # Complete containers (both apps and tasks finish) # Preempt containers # Nodes are down, this might not affect it directly, need to double check. Thanks Ray for pointing out. I could miss something. Correct me if you have anything else. I guess the test cases need to cover these. We might not need to worry about queue deletion since it seems not supported. YARN-4691 is about same thing for FSLeafQueue. Make sense to me to combine YARN-4691 in this JIRA. > Make Collections.sort() more efficient in FSParentQueue.java > > > Key: YARN-4090 > URL: https://issues.apache.org/jira/browse/YARN-4090 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Xianyin Xin >Assignee: zhangshilong > Attachments: sampling1.jpg, sampling2.jpg, YARN-4090.001.patch, > YARN-4090.002.patch, YARN-4090.003.patch, YARN-4090.004.patch, > YARN-4090.005.patch, YARN-4090.006.patch, YARN-4090-preview.patch, > YARN-4090-TestResult.pdf > > > Collections.sort() consumes too much time in a scheduling round. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5889) Improve and refactor user-limit calculation in capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859996#comment-15859996 ] Hudson commented on YARN-5889: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11227 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11227/]) YARN-5889. Improve and refactor user-limit calculation in Capacity (wangda: rev 5fb723bb77722d41df6959eee23e1b0cfeb5584e) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/FifoIntraQueuePreemptionPlugin.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ActiveUsersManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNodeLabelUpdate.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractUsersManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityHeadroomProvider.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicyMockFramework.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimitsByPartition.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Queue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UsersManager.java * (edit)
[jira] [Assigned] (YARN-6165) Intra-queue preemption occurs even when preemption is turned off for a specific queue.
[
https://issues.apache.org/jira/browse/YARN-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne reassigned YARN-6165:
Assignee: Eric Payne
> Intra-queue preemption occurs even when preemption is turned off for a
> specific queue.
> --
>
> Key: YARN-6165
> URL: https://issues.apache.org/jira/browse/YARN-6165
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.0.0-alpha2
>Reporter: Eric Payne
>Assignee: Eric Payne
>
> Intra-queue preemption occurs even when preemption is turned on for the whole
> cluster ({{yarn.resourcemanager.scheduler.monitor.enable == true}}) but
> turned off for a specific queue
> ({{yarn.scheduler.capacity.root.queue1.disable_preemption == true}}).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6165) Intra-queue preemption occurs even when preemption is turned off for a specific queue.
[
https://issues.apache.org/jira/browse/YARN-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859987#comment-15859987
]
Eric Payne commented on YARN-6165:
--
Use Case:
- Configure queues with cluster-wide preemption on, but a specific queue's
preemption off (see above).
- Submit a job at priority 1 that fills the entire queue
- Submit a job as the same user at priority 2
Containers from the first job will be preempted when they shouldn't be.
> Intra-queue preemption occurs even when preemption is turned off for a
> specific queue.
> --
>
> Key: YARN-6165
> URL: https://issues.apache.org/jira/browse/YARN-6165
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.0.0-alpha2
>Reporter: Eric Payne
>
> Intra-queue preemption occurs even when preemption is turned on for the whole
> cluster ({{yarn.resourcemanager.scheduler.monitor.enable == true}}) but
> turned off for a specific queue
> ({{yarn.scheduler.capacity.root.queue1.disable_preemption == true}}).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5889) Improve and refactor user-limit calculation in capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5889: - Summary: Improve and refactor user-limit calculation in capacity scheduler (was: Improve user-limit calculation in capacity scheduler) > Improve and refactor user-limit calculation in capacity scheduler > - > > Key: YARN-5889 > URL: https://issues.apache.org/jira/browse/YARN-5889 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-5889.0001.patch, > YARN-5889.0001.suggested.patchnotes, YARN-5889.0002.patch, > YARN-5889.0003.patch, YARN-5889.0004.patch, YARN-5889.0005.patch, > YARN-5889.0006.patch, YARN-5889.0007.patch, YARN-5889.0008.patch, > YARN-5889.0009.patch, YARN-5889.0010.patch, YARN-5889.v0.patch, > YARN-5889.v1.patch, YARN-5889.v2.patch > > > Currently user-limit is computed during every heartbeat allocation cycle with > a write lock. To improve performance, this tickets is focussing on moving > user-limit calculation out of heartbeat allocation flow. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6158) FairScheduler: app usage can go to negative
[
https://issues.apache.org/jira/browse/YARN-6158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated YARN-6158:
-
Attachment: YARN-6158.000.patch
Attaching patch.
> FairScheduler: app usage can go to negative
> ---
>
> Key: YARN-6158
> URL: https://issues.apache.org/jira/browse/YARN-6158
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6158.000.patch
>
>
> FiCaSchedulerApp.containerCompleted checks, if the container being completed
> is in the active list:
> {code}
> // Remove from the list of containers
> if (null == liveContainers.remove(containerId)) {
> return false;
> }
> {code}
> Fair scheduler should do the same, otherwise multiple different container
> close events leave the application with negative resource usage in
> {{queue.getMetrics().releaseResources}} and {{attemptResourceUsage.decUsed}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6165) Intra-queue preemption occurs even when preemption is turned off for a specific queue.
Eric Payne created YARN-6165:
Summary: Intra-queue preemption occurs even when preemption is
turned off for a specific queue.
Key: YARN-6165
URL: https://issues.apache.org/jira/browse/YARN-6165
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler, scheduler preemption
Affects Versions: 3.0.0-alpha2
Reporter: Eric Payne
Intra-queue preemption occurs even when preemption is turned on for the whole
cluster ({{yarn.resourcemanager.scheduler.monitor.enable == true}}) but turned
off for a specific queue
({{yarn.scheduler.capacity.root.queue1.disable_preemption == true}}).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859912#comment-15859912 ] Haibo Chen commented on YARN-4985: -- The dependency is not mainly because all tests are in the hbase-server module as they are now, but because FlowScanner needs to reference FlowRun table schema classes. If we were to extract another module that only includes table/column definitions, we can avoid the dependency from hbase-server on hbase-client. But hbase-client code seems quite coupled to the table schema code due to current code implementation (read & write methods are defined within the Table/Column classes). I will try to break the current hbase-client module in my patch into hbase-schema and hbase-client. This way, we should not have the undesirable dependency any more. That said, I think my previous concern about loading coprocessor from HDFS is still valid, since we will still have hbase-server depend on hbase-schema. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5700) testAMRestartNotLostContainerCompleteMsg times out intermittently in 2.8
[
https://issues.apache.org/jira/browse/YARN-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger resolved YARN-5700.
---
Resolution: Cannot Reproduce
Revisited this and I cannot reproduce the failure. Closing as cannot reproduce
> testAMRestartNotLostContainerCompleteMsg times out intermittently in 2.8
>
>
> Key: YARN-5700
> URL: https://issues.apache.org/jira/browse/YARN-5700
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
>
> {noformat}
> java.lang.Exception: test timed out after 3 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:301)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:286)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:281)
> at
> org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart.testAMRestartNotLostContainerCompleteMsg(TestAMRestart.java:774)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5956) Refactor ClientRMService
[
https://issues.apache.org/jira/browse/YARN-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859779#comment-15859779
]
Sunil G commented on YARN-5956:
---
Thanks [~lewuathe].
Were u mentioning about {{YARN-5956.08.patch}}. IN that patch also,
{{verifyUserAccessForRMApp}} internally calling {{checkAccess}}, correct? Am I
missing something.
> Refactor ClientRMService
>
>
> Key: YARN-5956
> URL: https://issues.apache.org/jira/browse/YARN-5956
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: YARN-5956.01.patch, YARN-5956.02.patch,
> YARN-5956.03.patch, YARN-5956.04.patch, YARN-5956.05.patch,
> YARN-5956.06.patch, YARN-5956.07.patch, YARN-5956.08.patch
>
>
> Some refactoring can be done in {{ClientRMService}}.
> - Remove redundant variable declaration
> - Fill in missing javadocs
> - Proper variable access modifier
> - Fix some typos in method name and exception messages
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5501) Container Pooling in YARN
[ https://issues.apache.org/jira/browse/YARN-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859587#comment-15859587 ] Jason Lowe commented on YARN-5501: -- Thanks for the detailed answers. I highly recommend these get captured in a new revision of the design document along with diagrams to help others come up to speed and join the discussion. Otherwise we have a wall of text in this JIRA that is essential reading for anyone understanding what is really being proposed. bq. If the app can provide some hints as to when it is good to consider a container pre-initialized then when the container finishes we can carry out the required operations to go back to the pre-init state. My thinking is the mere fact that the container finishes is indicative that it is ready for reuse. Maybe there's an explicit API they call at the end of the task or whatever, but the same has to be done for this existing design as well -- we need to know when the container is ready to be reused. The main difference I see in this approach is that there isn't an explicit 'pre-init' step where the users or admins need to premeditate what will be run. Instead the first run of the app framework is the same performance it is today, but subsequent runs are faster since it can reused those cached containers. Seems to me the most difficult part of this is coming up with an efficient container request protocol so YARN can know when it can reuse an old, cached container and when it cannot. The existing proposal works around this by requiring the containers to be setup beforehand as special resource types, but that won't work for a general container caching approach. bq. What you are saying makes sense, but in that case container resizing won't work as well. I don't believe it does for cases where we're trying to resize something like a JVM. I honestly don't know of any real-world use cases for it today, but my guess is that they either have a small front-end launcher that can re-launch something when they are told (via a completely separate, app-specific channel) that the resize is about to occur/has occurred, or they have explicit memory caching that they can trivially grow or shrink (again, completely outside of any help from YARN). bq. For our scenarios resource constraints are enforced via job objects or cgroups so things are ok. They certainly are enforced, but how does the app know about the new constraints so they can either avoid getting shot or take advantage of the new space? Simply updating the cgroup is not going to be sufficient. Either the process will OOM because it slams into the new lower limit (potentially instantly if it is already bigger than the new limit) or it will be completely oblivious that it now has acres of memory that it can use. If it tried to use it before it would fail, so how does it know it grew? For example, the JVM can't magically do this unless the app is doing some sort of explicit off-heap memory management via direct buffers, etc. and is told about its memory limit. Simply updating the cgroup setting doesn't seem to be a sufficient communication channel here, so I'm curious how that's all you need to do for your scenario. It seems to me there needs to be a predictable sequence here, and it's not the same for growing and shrinking. When we are growing the memory of a container it's a bit simpler. The container needs to be signaled about the resizing event _after_ the resizing has occurred at the enforcement level (i.e.: cgroup). This could simply be part of the code that the application runs as part of reusing the container. However if we are shrinking the memory of the container then we need to signal the container _before_ the resizing has occurred _and_ we need to know when the container process(es) have had a chance to react to that signal and get under the new, lower limit before actually enforcing it (i..e: at the cgroup level). Otherwise the only way I can see this working is if the container resizing isn't generalized but requires the container code to always shrink itself to a minimal acceptable level (i.e.: the smallest that will ever be requested) after running each app's task so every case essentially becomes the container growth scenario. > Container Pooling in YARN > - > > Key: YARN-5501 > URL: https://issues.apache.org/jira/browse/YARN-5501 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun Suresh >Assignee: Hitesh Sharma > Attachments: Container Pooling - one pager.pdf > > > This JIRA proposes a method for reducing the container launch latency in > YARN. It introduces a notion of pooling *Unattached Pre-Initialized > Containers*. > Proposal in brief: > * Have a *Pre-Initialized Container Factory* service within the NM to
[jira] [Commented] (YARN-5956) Refactor ClientRMService
[
https://issues.apache.org/jira/browse/YARN-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859464#comment-15859464
]
Kai Sasaki commented on YARN-5956:
--
[~sunilg] Sorry for late. I checked redundant {{checkAccess}} call and updated.
Failed tests seems not related to the patch. I confirmed it passed at local.
Could you check again please?
> Refactor ClientRMService
>
>
> Key: YARN-5956
> URL: https://issues.apache.org/jira/browse/YARN-5956
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: YARN-5956.01.patch, YARN-5956.02.patch,
> YARN-5956.03.patch, YARN-5956.04.patch, YARN-5956.05.patch,
> YARN-5956.06.patch, YARN-5956.07.patch, YARN-5956.08.patch
>
>
> Some refactoring can be done in {{ClientRMService}}.
> - Remove redundant variable declaration
> - Fill in missing javadocs
> - Proper variable access modifier
> - Fix some typos in method name and exception messages
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6113) re-direct NM Web Service to get container logs for finished applications
[ https://issues.apache.org/jira/browse/YARN-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859421#comment-15859421 ] Junping Du commented on YARN-6113: -- Thanks Xuan for updating the patch. Latest patch LGTM. However, does UT failures related to patch here? I cannot reproduce the failure w/o patch locally. > re-direct NM Web Service to get container logs for finished applications > > > Key: YARN-6113 > URL: https://issues.apache.org/jira/browse/YARN-6113 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-6113.branch-2.v1.patch, > YARN-6113.branch-2.v2.patch, YARN-6113.branch-2.v3.patch, > YARN-6113.branch-2.v4.patch, YARN-6113.trunk.v2.patch, > YARN-6113.trunk.v3.patch, YARN-6113.trunk.v4.patch > > > In NM web ui, when we try to get container logs for finished application, it > would redirect to the log server based on the configuration: > yarn.log.server.url. We should do the similar thing for NM WebService -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Improve /flows API for more flexible filters fromid, collapse, userid
[ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859344#comment-15859344 ] Varun Saxena commented on YARN-6027: Ok...I had forgotton that we currently do not limit flow runs which means that in collapse mode, if daterange is not specified, we will have to do a full table scan anyways. I have a JIRA for limiting flowruns. I will discuss with some HBase guy and check the scenario with him and handle it in that JIRA, if required. Basically its a choice between a full table scan of say 10 rows vs multiple scans with PageFilter equal to limit in each scan(or 2-3 times the limit value) and breaking out from the loop as soon as flow run limit is also reached. By default we can return 0 flow runs for flows endpoint as we already have a flowruns endpoint as well. Anyways I am not a 100% sure on this and need to check with a HBase guy. We can get back to this on the JIRA which is with me. > Improve /flows API for more flexible filters fromid, collapse, userid > - > > Key: YARN-6027 > URL: https://issues.apache.org/jira/browse/YARN-6027 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6027-YARN-5355.0001.patch > > > In YARN-5585 , fromId is supported for retrieving entities. We need similar > filter for flows/flowRun apps and flow run and flow as well. > Along with supporting fromId, this JIRA should also discuss following points > * Should we throw an exception for entities/entity retrieval if duplicates > found? > * TimelieEntity : > ** Should equals method also check for idPrefix? > ** Does idPrefix is part of identifiers? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6157) Inconsistencies in verifying Max Applications
[
https://issues.apache.org/jira/browse/YARN-6157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859205#comment-15859205
]
Varun Saxena commented on YARN-6157:
[~Naganarasimha], the check for system max apps is made in
CapacityScheduler#addApplication and flow for recovered apps goes via
CapacityScheduler#addApplicationOnRecovery. So recovered apps I think will be
excluded.
Other point is right. We probably need a counter in LeafQueue which will be
increment when we call LeafQueue#submitApplication. We have a similar counter
in ParentQueue too.
> Inconsistencies in verifying Max Applications
> -
>
> Key: YARN-6157
> URL: https://issues.apache.org/jira/browse/YARN-6157
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>
> Inconsistencies in verifying Max Applications when the max apps is reduced
> and either HA is done ow work preserving restart is done.
> # currently Max applications across cluster should not be done for the
> recovered apps. Seems like currently we are doing it
> # Max applications for a queue is done @ CapacityScheduler.addApplication
> which considers sum of Pending and running applications but we add to pending
> applications in {{CapacityScheduler.addApplicationAttempt ->
> LeafQueue.addApplicationAttempt}} so between these 2 checks we can activate
> more apps than what can queue restrict.
> # During recovery of a RMApp, if applicationAttempts are not found then we
> recover it without recovery false @ {{RMAppImpl.RMAppRecoveredTransition}},
> this can lead to failure of apps which were accepted earlier but attempt was
> not yet created and HA happens when MAX app configuration (for cluster/queue)
> is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
