[jira] [Commented] (YARN-10472) Backport YARN-10314 to branch-3.2
[ https://issues.apache.org/jira/browse/YARN-10472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223431#comment-17223431 ] Xiaoqiao He commented on YARN-10472: Thanks [~smeng], I would like to do cherry-pick them together shortly. Thanks > Backport YARN-10314 to branch-3.2 > - > > Key: YARN-10472 > URL: https://issues.apache.org/jira/browse/YARN-10472 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.2.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Blocker > Fix For: 3.2.3 > > > Filing this jira to raise the following concern: > YARN-10314 fixes a problem with the shaded jars in 3.3.0. But it is not > backported to branch-3.2 yet. [~weichiu] and I ([~smeng]) are looking into > this. > I have submitted a PR on branch-3.2: > https://github.com/apache/hadoop/pull/2412 > CC [~hexiaoqiao] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10475) Scale RM-NM heartbeat interval based on node utilization
[
https://issues.apache.org/jira/browse/YARN-10475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223343#comment-17223343
]
Hadoop QA commented on YARN-10475:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
28s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | | {color:blue} markdownlint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 3 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
24s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
10s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
55s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
42s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
47s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
23m 12s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
9s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
13s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
53s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | | {color:blue}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no findbugs output file
(findbugsXml.xml) {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
43s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
44s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
44s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
52s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
52s{color} | | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} blanks {color} | {color:red} 0m
0s{color} |
[/blanks-eol.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/273/artifact/out/blanks-eol.txt]
| {color:red} The patch has 1 line(s) that end in blanks. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 41s{color} |
[/results-checkstyle-root.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/273/artifact/out/results-checkstyle-root.txt]
| {color:orange} root: The patch generated 3 new + 412 unchanged - 0 fixed =
415 total (was 412) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
48s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} xml {color}
[jira] [Commented] (YARN-10472) Backport YARN-10314 to branch-3.2
[ https://issues.apache.org/jira/browse/YARN-10472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223270#comment-17223270 ] Siyao Meng commented on YARN-10472: --- Thanks [~hexiaoqiao]. I've merged it to branch-3.2. But it looks like [3.2.2|https://github.com/apache/hadoop/commits/branch-3.2.2] is branched from [3.2|https://github.com/apache/hadoop/commits/branch-3.2] already. The former branch is missing a dozen or so commits so far. Should I manually cherry-pick the fix to branch-3.2.2 on my side? or you could do some magic on your end. your choice. let me know. :D > Backport YARN-10314 to branch-3.2 > - > > Key: YARN-10472 > URL: https://issues.apache.org/jira/browse/YARN-10472 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.2.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Blocker > Fix For: 3.2.3 > > > Filing this jira to raise the following concern: > YARN-10314 fixes a problem with the shaded jars in 3.3.0. But it is not > backported to branch-3.2 yet. [~weichiu] and I ([~smeng]) are looking into > this. > I have submitted a PR on branch-3.2: > https://github.com/apache/hadoop/pull/2412 > CC [~hexiaoqiao] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10472) Backport YARN-10314 to branch-3.2
[ https://issues.apache.org/jira/browse/YARN-10472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated YARN-10472: -- Fix Version/s: 3.2.3 > Backport YARN-10314 to branch-3.2 > - > > Key: YARN-10472 > URL: https://issues.apache.org/jira/browse/YARN-10472 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.2.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Blocker > Fix For: 3.2.3 > > > Filing this jira to raise the following concern: > YARN-10314 fixes a problem with the shaded jars in 3.3.0. But it is not > backported to branch-3.2 yet. [~weichiu] and I ([~smeng]) are looking into > this. > I have submitted a PR on branch-3.2: > https://github.com/apache/hadoop/pull/2412 > CC [~hexiaoqiao] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10475) Scale RM-NM heartbeat interval based on node utilization
[ https://issues.apache.org/jira/browse/YARN-10475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223217#comment-17223217 ] Jim Brennan commented on YARN-10475: Thanks [~epayne]! I put up patch 003, which adds documentation to Nodemanager.md and also fixes a minor typo in yarn-default.xml. > Scale RM-NM heartbeat interval based on node utilization > > > Key: YARN-10475 > URL: https://issues.apache.org/jira/browse/YARN-10475 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 2.10.1, 3.4.1 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-10475.001.patch, YARN-10475.002.patch, > YARN-10475.003.patch > > > Add the ability to scale the RM-NM heartbeat interval based on node cpu > utilization compared to overall cluster cpu utilization. If a node is > over-utilized compared to the rest of the cluster, it's heartbeat interval > slows down. If it is under-utilized compared to the rest of the cluster, > it's heartbeat interval speeds up. > This is a feature we have been running with internally in production for > several years. It was developed by [~nroberts], based on the observation > that larger faster nodes on our cluster were under-utilized compared to > smaller slower nodes. > This feature is dependent on [YARN-10450], which added cluster-wide > utilization metrics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10475) Scale RM-NM heartbeat interval based on node utilization
[ https://issues.apache.org/jira/browse/YARN-10475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan updated YARN-10475: --- Attachment: YARN-10475.003.patch > Scale RM-NM heartbeat interval based on node utilization > > > Key: YARN-10475 > URL: https://issues.apache.org/jira/browse/YARN-10475 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 2.10.1, 3.4.1 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-10475.001.patch, YARN-10475.002.patch, > YARN-10475.003.patch > > > Add the ability to scale the RM-NM heartbeat interval based on node cpu > utilization compared to overall cluster cpu utilization. If a node is > over-utilized compared to the rest of the cluster, it's heartbeat interval > slows down. If it is under-utilized compared to the rest of the cluster, > it's heartbeat interval speeds up. > This is a feature we have been running with internally in production for > several years. It was developed by [~nroberts], based on the observation > that larger faster nodes on our cluster were under-utilized compared to > smaller slower nodes. > This feature is dependent on [YARN-10450], which added cluster-wide > utilization metrics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10471) Prevent logs for any container from becoming larger than a configurable size.
[ https://issues.apache.org/jira/browse/YARN-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223200#comment-17223200 ] Eric Payne commented on YARN-10471: --- Thanks a lot, [~Jim_Brennan]! I don't think it's necessary to port this back to 3.1 or 2.10. What do you think? > Prevent logs for any container from becoming larger than a configurable size. > - > > Key: YARN-10471 > URL: https://issues.apache.org/jira/browse/YARN-10471 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.2.1, 3.1.4 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Minor > Fix For: 3.3.1, 3.4.1, 3.2.3 > > Attachments: YARN.10471.001.patch, YARN.10471.002.patch, > YARN.10471.003.patch, YARN.10471.004.patch, YARN.10471.005.patch, > YARN.10471.branch-3.2.003.patch, YARN.10471.branch-3.2.005.patch > > > Configure a cluster such that a task attempt will be killed if any container > log exceeds a configured size. This would help prevent logs from filling > disks and also prevent the need to aggregate enormous logs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10475) Scale RM-NM heartbeat interval based on node utilization
[ https://issues.apache.org/jira/browse/YARN-10475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223196#comment-17223196 ] Eric Payne commented on YARN-10475: --- [~Jim_Brennan], Thanks for working on this feature and providing the patch. The code patch looks good to me. Once you provide the documentation of the new properties, I am ready to provide my +1. > Scale RM-NM heartbeat interval based on node utilization > > > Key: YARN-10475 > URL: https://issues.apache.org/jira/browse/YARN-10475 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 2.10.1, 3.4.1 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-10475.001.patch, YARN-10475.002.patch > > > Add the ability to scale the RM-NM heartbeat interval based on node cpu > utilization compared to overall cluster cpu utilization. If a node is > over-utilized compared to the rest of the cluster, it's heartbeat interval > slows down. If it is under-utilized compared to the rest of the cluster, > it's heartbeat interval speeds up. > This is a feature we have been running with internally in production for > several years. It was developed by [~nroberts], based on the observation > that larger faster nodes on our cluster were under-utilized compared to > smaller slower nodes. > This feature is dependent on [YARN-10450], which added cluster-wide > utilization metrics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10471) Prevent logs for any container from becoming larger than a configurable size.
[ https://issues.apache.org/jira/browse/YARN-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223195#comment-17223195 ] Jim Brennan commented on YARN-10471: Thanks [~epayne]! I have committed this to trunk, branch-3.3, and branch-3.2. The branch-3.2 patch does not apply to branch-3.1 nor branch-2.10. Please provide patches for those branches if you want this committed further back. > Prevent logs for any container from becoming larger than a configurable size. > - > > Key: YARN-10471 > URL: https://issues.apache.org/jira/browse/YARN-10471 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.2.1, 3.1.4 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Minor > Fix For: 3.3.1, 3.4.1, 3.2.3 > > Attachments: YARN.10471.001.patch, YARN.10471.002.patch, > YARN.10471.003.patch, YARN.10471.004.patch, YARN.10471.005.patch, > YARN.10471.branch-3.2.003.patch, YARN.10471.branch-3.2.005.patch > > > Configure a cluster such that a task attempt will be killed if any container > log exceeds a configured size. This would help prevent logs from filling > disks and also prevent the need to aggregate enormous logs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10471) Prevent logs for any container from becoming larger than a configurable size.
[ https://issues.apache.org/jira/browse/YARN-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan updated YARN-10471: --- Fix Version/s: 3.2.3 3.4.1 3.3.1 > Prevent logs for any container from becoming larger than a configurable size. > - > > Key: YARN-10471 > URL: https://issues.apache.org/jira/browse/YARN-10471 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.2.1, 3.1.4 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Minor > Fix For: 3.3.1, 3.4.1, 3.2.3 > > Attachments: YARN.10471.001.patch, YARN.10471.002.patch, > YARN.10471.003.patch, YARN.10471.004.patch, YARN.10471.005.patch, > YARN.10471.branch-3.2.003.patch, YARN.10471.branch-3.2.005.patch > > > Configure a cluster such that a task attempt will be killed if any container > log exceeds a configured size. This would help prevent logs from filling > disks and also prevent the need to aggregate enormous logs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10471) Prevent logs for any container from becoming larger than a configurable size.
[
https://issues.apache.org/jira/browse/YARN-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223179#comment-17223179
]
Hadoop QA commented on YARN-10471:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
58s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | | {color:blue} markdownlint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
21s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
4s{color} | | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
41s{color} | | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
14s{color} | | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 54s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | | {color:green} branch-3.2 passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
34s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | | {color:blue}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no findbugs output file
(findbugsXml.xml) {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
55s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
55s{color} | | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} blanks {color} | {color:red} 0m
0s{color} |
[/blanks-tabs.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/272/artifact/out/blanks-tabs.txt]
| {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
3s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 26s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
41s{color} | | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | | {color:blue} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
has no data from findbugs {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
16s{color} | | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 48s{color}
|
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-
[jira] [Commented] (YARN-10476) Queue metrics for Unmanaged applications
[
https://issues.apache.org/jira/browse/YARN-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223165#comment-17223165
]
Hadoop QA commented on YARN-10476:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
16s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 3 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
23s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 14s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
2s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/271/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt]
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 8 new + 457 unchanged - 1 fixed = 465 total (was 458)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 25s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | | {color:green} the patch passed with JDK Private
[jira] [Commented] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
[
https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223117#comment-17223117
]
Wangda Tan commented on YARN-10458:
---
[~pbacsko], there're two issues in the test, one is setup NodelabelManager
after RM created, it somehow didn't get the right label manager (I didn't do
further troubleshooting), the correct way to do it is:
{code:java}
MockRM rm = new MockRM(csConf) {
@Override
public RMNodeLabelsManager createNodeLabelManager() {
return mgr;
}
}; {code}
The label manager is used by scheduler to correctly calculate effective
resources:
*org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AutoCreatedLeafQueue#mergeCapacities*
{code:java}
Resource resourceByLabel = labelManager.getResourceByLabel(nodeLabel,
csContext.getClusterResource()); {code}
So it causes the app cannot move to RUNNING because effective is always 0.
Second issue is, you called a nodeHearbeat before launchAndRegisterAM, it makes
app attempt advanced to ALLOCATED state instead of SCHEDULED state. after
removed the hearbeat call, it works fine now.
Please add more checks for the queue creation, and I suggest to move this test
to TestCapacitySchedulerAutoQUeueCreation.
A good resource to reference to is tests inside
`TestNodeLabelContainerAllocation` if you write a test, reference to
TestNodeLabelContainerAllocation will be a good starting point.
Here's full test code after changes I made:
{code:java}
@Test
public void testAccessCheckOfNonExistingDynamicQueueWithTags()
throws Exception {
CapacitySchedulerConfiguration csConf
= new CapacitySchedulerConfiguration();
csConf.setQueues(CapacitySchedulerConfiguration.ROOT,
new String[] {"a", "b"});
csConf.setCapacity("root.a", 90);
csConf.setCapacity("root.b", 10);
csConf.set("yarn.scheduler.capacity.resource-calculator",
"org.apache.hadoop.yarn.util.resource.DominantResourceCalculator");
csConf.setAutoCreateChildQueueEnabled("root.a", true);
csConf.setAutoCreatedLeafQueueConfigCapacity("root.a", 50);
csConf.setAutoCreatedLeafQueueConfigMaxCapacity("root.a", 100);
csConf.set(CapacitySchedulerConfiguration.MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT,
"0.5");
csConf.setAcl("root.a", QueueACL.ADMINISTER_QUEUE, "*");
csConf.setAcl("root.a", QueueACL.SUBMIT_APPLICATIONS, "*");
csConf.setBoolean(YarnConfiguration
.APPLICATION_TAG_BASED_PLACEMENT_ENABLED, true);
csConf.setStrings(YarnConfiguration
.APPLICATION_TAG_BASED_PLACEMENT_USER_WHITELIST, "hadoop");
csConf.set(CapacitySchedulerConfiguration.QUEUE_MAPPING,
"u:%user:root.a.%user");
csConf.setInt("yarn.scheduler.minimum-allocation-mb", 1024);
csConf.setInt("yarn.scheduler.minimum-allocation-vcores", 1);
YarnConfiguration conf=new YarnConfiguration(csConf);
conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
ResourceScheduler.class);
RMNodeLabelsManager mgr = new NullRMNodeLabelsManager();
mgr.init(conf);
MockRM rm = new MockRM(csConf) {
@Override
public RMNodeLabelsManager createNodeLabelManager() {
return mgr;
}
};
rm.start();
MockNM nm = rm.registerNode("127.0.0.1:1234", 16 * GB);
MockRMAppSubmissionData data =
MockRMAppSubmissionData.Builder.createWithMemory(GB, rm)
.withAppName("apptodynamicqueue")
.withUser("hadoop")
.withAcls(null)
.withUnmanagedAM(false)
.withApplicationTags(Sets.newHashSet("userid=testuser"))
.build();
RMApp app = MockRMAppSubmitter.submit(rm, data);
MockRM.launchAndRegisterAM(app, rm, nm); // stuck in SCHEDULED state
nm.nodeHeartbeat(true);
}{code}
> Hive On Tez queries fails upon submission to dynamically created pools
> --
>
> Key: YARN-10458
> URL: https://issues.apache.org/jira/browse/YARN-10458
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Anand Srinivasan
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10458-001.patch, YARN-10458-002.patch
>
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see
> that the queue creation fails because ACL submit application check couldn't
> succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues.
> For static queues, this worked but failed for dynamic queues. Also tried
> setting the below property but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
> 2020-09-18 01:08:40,579 INFO
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
> Application application_1600399068816_0460 user user1
[jira] [Updated] (YARN-10471) Prevent logs for any container from becoming larger than a configurable size.
[ https://issues.apache.org/jira/browse/YARN-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-10471: -- Attachment: YARN.10471.branch-3.2.005.patch > Prevent logs for any container from becoming larger than a configurable size. > - > > Key: YARN-10471 > URL: https://issues.apache.org/jira/browse/YARN-10471 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.2.1, 3.1.4 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Minor > Attachments: YARN.10471.001.patch, YARN.10471.002.patch, > YARN.10471.003.patch, YARN.10471.004.patch, YARN.10471.005.patch, > YARN.10471.branch-3.2.003.patch, YARN.10471.branch-3.2.005.patch > > > Configure a cluster such that a task attempt will be killed if any container > log exceeds a configured size. This would help prevent logs from filling > disks and also prevent the need to aggregate enormous logs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10425) Replace the legacy placement engine in CS with the new one
[
https://issues.apache.org/jira/browse/YARN-10425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223045#comment-17223045
]
Hadoop QA commented on YARN-10425:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 9 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
34s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 7s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
44s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/270/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt]
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 7 new + 301 unchanged - 8 fixed = 308 total (was 309)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 17s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | | {color:green} the patch passed with JDK Priva
[jira] [Commented] (YARN-10476) Queue metrics for Unmanaged applications
[ https://issues.apache.org/jira/browse/YARN-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223011#comment-17223011 ] Cyrus Jackson commented on YARN-10476: -- Instead of using rmContext to get the unmanagedAM as its an overkill. In patch 2, I'm getting unmanagedAM flag from scheduler side itself. > Queue metrics for Unmanaged applications > - > > Key: YARN-10476 > URL: https://issues.apache.org/jira/browse/YARN-10476 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Cyrus Jackson >Assignee: Cyrus Jackson >Priority: Minor > Attachments: YARN-10476.001.patch, YARN-10476.002.patch > > > Right now we do not have separate metrics unmanaged applications. All > application metrics come as part of Queue (Managed and UnManaged), This Jira > aims to show them separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10476) Queue metrics for Unmanaged applications
[ https://issues.apache.org/jira/browse/YARN-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyrus Jackson updated YARN-10476: - Attachment: YARN-10476.002.patch > Queue metrics for Unmanaged applications > - > > Key: YARN-10476 > URL: https://issues.apache.org/jira/browse/YARN-10476 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Cyrus Jackson >Assignee: Cyrus Jackson >Priority: Minor > Attachments: YARN-10476.001.patch, YARN-10476.002.patch > > > Right now we do not have separate metrics unmanaged applications. All > application metrics come as part of Queue (Managed and UnManaged), This Jira > aims to show them separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10476) Queue metrics for Unmanaged applications
[
https://issues.apache.org/jira/browse/YARN-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222934#comment-17222934
]
Hadoop QA commented on YARN-10476:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
57s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | | {color:red} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
12s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 10s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
4s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
1s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/269/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt]
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 6 new + 334 unchanged - 0 fixed = 340 total (was 334)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 40s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
38s{color} |
[/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUb
[jira] [Commented] (YARN-10425) Replace the legacy placement engine in CS with the new one
[ https://issues.apache.org/jira/browse/YARN-10425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222918#comment-17222918 ] Gergely Pollak commented on YARN-10425: --- Patch#4 is about bugfixes again, this time probably all of them are fixed and can move onto the actual review feedbacks. > Replace the legacy placement engine in CS with the new one > -- > > Key: YARN-10425 > URL: https://issues.apache.org/jira/browse/YARN-10425 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10425.001.patch, YARN-10425.002.patch, > YARN-10425.003.patch, YARN-10425.004.patch > > > Remove the UserGroupMapping and ApplicationName mapping classes, and use the > new CSMappingPlacementRule instead. Also cleanup the orphan classes which are > used by these classes only. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10425) Replace the legacy placement engine in CS with the new one
[ https://issues.apache.org/jira/browse/YARN-10425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak updated YARN-10425: -- Attachment: YARN-10425.004.patch > Replace the legacy placement engine in CS with the new one > -- > > Key: YARN-10425 > URL: https://issues.apache.org/jira/browse/YARN-10425 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10425.001.patch, YARN-10425.002.patch, > YARN-10425.003.patch, YARN-10425.004.patch > > > Remove the UserGroupMapping and ApplicationName mapping classes, and use the > new CSMappingPlacementRule instead. Also cleanup the orphan classes which are > used by these classes only. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
[
https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222860#comment-17222860
]
Peter Bacsko edited comment on YARN-10458 at 10/29/20, 12:24 PM:
-
[~wangda] I created a test case with MockRM and MockNM but I have a little bit
of a problem. For some reason, the submitted application doesn't reach
ALLOCATED state, it' stuck in SCHEDULED. I tried to dig deeper but got confused
about all kinds of resource calculations.
The access check passes, so there's no problem there, but why can't the
application start?
Here is the testcase which I added to {{TestCapacityScheduler.java}}
{noformat}
import org.apache.hadoop.yarn.api.records.QueueACL;
...
@Test
public void testAccessCheckOfNonExistingDynamicQueueWithTags()
throws Exception {
CapacitySchedulerConfiguration csConf
= new CapacitySchedulerConfiguration();
csConf.setQueues(CapacitySchedulerConfiguration.ROOT,
new String[] {"a", "b"});
csConf.setCapacity("root.a", 90);
csConf.setCapacity("root.b", 10);
csConf.set("yarn.scheduler.capacity.resource-calculator",
"org.apache.hadoop.yarn.util.resource.DominantResourceCalculator");
csConf.setAutoCreateChildQueueEnabled("root.a", true);
csConf.setAutoCreatedLeafQueueConfigCapacity("root.a", 50);
csConf.setAutoCreatedLeafQueueConfigMaxCapacity("root.a", 100);
csConf.set(CapacitySchedulerConfiguration.MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT,
"0.5");
csConf.setAcl("root.a", QueueACL.ADMINISTER_QUEUE, "*");
csConf.setAcl("root.a", QueueACL.SUBMIT_APPLICATIONS, "*");
csConf.setBoolean(YarnConfiguration
.APPLICATION_TAG_BASED_PLACEMENT_ENABLED, true);
csConf.setStrings(YarnConfiguration
.APPLICATION_TAG_BASED_PLACEMENT_USER_WHITELIST, "hadoop");
csConf.set(CapacitySchedulerConfiguration.QUEUE_MAPPING,
"u:%user:root.a.%user");
csConf.setInt("yarn.scheduler.minimum-allocation-mb", 1024);
csConf.setInt("yarn.scheduler.minimum-allocation-vcores", 1);
YarnConfiguration conf=new YarnConfiguration(csConf);
conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
ResourceScheduler.class);
RMNodeLabelsManager mgr=new NullRMNodeLabelsManager();
mgr.init(conf);
MockRM rm = new MockRM(csConf);
rm.getRMContext().setNodeLabelManager(mgr);
rm.start();
MockNM nm = rm.registerNode("127.0.0.1:1234", 16 * GB);
MockRMAppSubmissionData data =
MockRMAppSubmissionData.Builder.createWithMemory(GB, rm)
.withAppName("apptodynamicqueue")
.withUser("hadoop")
.withAcls(null)
.withUnmanagedAM(false)
.withApplicationTags(Sets.newHashSet("userid=testuser"))
.build();
RMApp app = MockRMAppSubmitter.submit(rm, data);
nm.nodeHeartbeat(true);
MockRM.launchAndRegisterAM(app, rm, nm); // stuck in SCHEDULED state
}
{noformat}
As you can see, the mapped queue becomes "root.a.testuser" and it gets created
but can't run applications:
{noformat}
2020-10-29 11:38:57,334 DEBUG [AsyncDispatcher event handler]
capacity.ParentQueue (ParentQueue.java:printChildQueues(861)) -
printChildQueues - queue: root.a child-queues:
root.a.testuserusedCapacity=(0.0), label=(*)
2020-10-29 11:38:57,335 DEBUG [AsyncDispatcher event handler]
capacity.ParentQueue (ParentQueue.java:assignContainersToChildQueues(799)) -
Trying to assign to queue: root.a.testuser stats: root.a.testuser:
capacity=0.5, absoluteCapacity=0.45, usedResources=,
usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0,
effectiveMinResource= , effectiveMaxResource=
2020-10-29 11:38:57,335 DEBUG [AsyncDispatcher event handler]
capacity.LeafQueue (LeafQueue.java:assignContainers(1129)) - assignContainers:
partition= #applications=1
2020-10-29 11:38:57,339 DEBUG [AsyncDispatcher event handler]
capacity.AbstractCSQueue (AbstractCSQueue.java:canAssignToThisQueue(1113)) -
Failed to assign to queue: root.a.testuser nodePatrition: , usedResources:
, clusterResources: ,
reservedResources: , maxLimitCapacity: , currTotalUsed:
{noformat}
I'm totally exhausted by this. Maybe the solution is obvious but I just can't
see it. The issue seems to be with effectiveMaxResource which is always
{{}} and a comparison fails in
{{AbstractCSQueue.java:canAssignToThisQueue()}}.
was (Author: pbacsko):
[~wangda] I created a test case with MockRM and MockNM but I have a little bit
of a problem. For some reason, the submitted application doesn't reach RUNNING
state, it' stuck in SCHEDULED. I tried to dig deeper but got confused about all
kinds of resource calculations.
The access check passes, so there's no problem there, but why can't the
application start?
Here is the testcase which I added to {{TestCapacityScheduler.java}}
{noformat}
import org.apache.ha
[jira] [Commented] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
[
https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222860#comment-17222860
]
Peter Bacsko commented on YARN-10458:
-
[~wangda] I created a test case with MockRM and MockNM but I have a little bit
of a problem. For some reason, the submitted application doesn't reach RUNNING
state, it' stuck in SCHEDULED. I tried to dig deeper but got confused about all
kinds of resource calculations.
The access check passes, so there's no problem there, but why can't the
application start?
Here is the testcase which I added to {{TestCapacityScheduler.java}}
{noformat}
import org.apache.hadoop.yarn.api.records.QueueACL;
...
@Test
public void testAccessCheckOfNonExistingDynamicQueueWithTags()
throws Exception {
CapacitySchedulerConfiguration csConf
= new CapacitySchedulerConfiguration();
csConf.setQueues(CapacitySchedulerConfiguration.ROOT,
new String[] {"a", "b"});
csConf.setCapacity("root.a", 90);
csConf.setCapacity("root.b", 10);
csConf.set("yarn.scheduler.capacity.resource-calculator",
"org.apache.hadoop.yarn.util.resource.DominantResourceCalculator");
csConf.setAutoCreateChildQueueEnabled("root.a", true);
csConf.setAutoCreatedLeafQueueConfigCapacity("root.a", 50);
csConf.setAutoCreatedLeafQueueConfigMaxCapacity("root.a", 100);
csConf.set(CapacitySchedulerConfiguration.MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT,
"0.5");
csConf.setAcl("root.a", QueueACL.ADMINISTER_QUEUE, "*");
csConf.setAcl("root.a", QueueACL.SUBMIT_APPLICATIONS, "*");
csConf.setBoolean(YarnConfiguration
.APPLICATION_TAG_BASED_PLACEMENT_ENABLED, true);
csConf.setStrings(YarnConfiguration
.APPLICATION_TAG_BASED_PLACEMENT_USER_WHITELIST, "hadoop");
csConf.set(CapacitySchedulerConfiguration.QUEUE_MAPPING,
"u:%user:root.a.%user");
csConf.setInt("yarn.scheduler.minimum-allocation-mb", 1024);
csConf.setInt("yarn.scheduler.minimum-allocation-vcores", 1);
YarnConfiguration conf=new YarnConfiguration(csConf);
conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
ResourceScheduler.class);
RMNodeLabelsManager mgr=new NullRMNodeLabelsManager();
mgr.init(conf);
MockRM rm = new MockRM(csConf);
rm.getRMContext().setNodeLabelManager(mgr);
rm.start();
MockNM nm = rm.registerNode("127.0.0.1:1234", 16 * GB);
MockRMAppSubmissionData data =
MockRMAppSubmissionData.Builder.createWithMemory(GB, rm)
.withAppName("apptodynamicqueue")
.withUser("hadoop")
.withAcls(null)
.withUnmanagedAM(false)
.withApplicationTags(Sets.newHashSet("userid=testuser"))
.build();
RMApp app = MockRMAppSubmitter.submit(rm, data);
nm.nodeHeartbeat(true);
MockRM.launchAndRegisterAM(app, rm, nm); // stuck in SCHEDULED state
}
{noformat}
As you can see, the mapped queue becomes "root.a.testuser" and it gets created
but can't run applications.
> Hive On Tez queries fails upon submission to dynamically created pools
> --
>
> Key: YARN-10458
> URL: https://issues.apache.org/jira/browse/YARN-10458
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Anand Srinivasan
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10458-001.patch, YARN-10458-002.patch
>
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see
> that the queue creation fails because ACL submit application check couldn't
> succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues.
> For static queues, this worked but failed for dynamic queues. Also tried
> setting the below property but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
> 2020-09-18 01:08:40,579 INFO
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
> Application application_1600399068816_0460 user user1 mapping [default] to
> [queue1] override false
> 2020-09-18 01:08:40,579 WARN
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from
> application tag does not have access to queue 'user1'. The placement is done
> for user 'hive'
>
> Checking the code, scheduler#checkAccess() bails out even before checking the
> ACL permissions for that particular queue because the CSQueue is null.
> {code:java}
> public boolean checkAccess(UserGroupInformation callerUGI,
> QueueACL acl, String queueName) {
> CSQueue queue = getQueue(queueName);
> if (queue == null) {
> if (LOG.isDebugEnabled())
> { LOG.debug("ACL not found for queue
[jira] [Updated] (YARN-10476) Queue metrics for Unmanaged applications
[ https://issues.apache.org/jira/browse/YARN-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyrus Jackson updated YARN-10476: - Attachment: (was: YARN-10476.001.patch) > Queue metrics for Unmanaged applications > - > > Key: YARN-10476 > URL: https://issues.apache.org/jira/browse/YARN-10476 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Cyrus Jackson >Assignee: Cyrus Jackson >Priority: Minor > Attachments: YARN-10476.001.patch > > > Right now we do not have separate metrics unmanaged applications. All > application metrics come as part of Queue (Managed and UnManaged), This Jira > aims to show them separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10476) Queue metrics for Unmanaged applications
[ https://issues.apache.org/jira/browse/YARN-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyrus Jackson updated YARN-10476: - Summary: Queue metrics for Unmanaged applications (was: Queue metrics of Unmanaged applications) > Queue metrics for Unmanaged applications > - > > Key: YARN-10476 > URL: https://issues.apache.org/jira/browse/YARN-10476 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Cyrus Jackson >Assignee: Cyrus Jackson >Priority: Minor > Attachments: YARN-10476.001.patch > > > Right now we do not have separate metrics unmanaged applications. All > application metrics come as part of Queue (Managed and UnManaged), This Jira > aims to show them separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
