[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level
[
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939818#comment-16939818
]
Jonathan Hung commented on YARN-9760:
-
Thanks [~varun_saxena]. Seems fine. A couple nits:
* Do we need to check in
{{hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/capacity-scheduler.xml}}
?
* The YarnConfiguration comment is a bit confusing:
{noformat}
// Configuration for the prefix of the tag which contains workflow ID,
// followed by the prefix. {noformat}
Can we reword this? Maybe "Prefix of an application tag which is followed by
that application's workflow id"
* There's a stray "{{as the user."}} line in the description for
yarn.scheduler.capacity.workflow-priority-mappings (in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf/capacity-scheduler.xml)
* Can we make CapacitySchedulerConfiguration#getWorkflowPriorityMappings
return Collection instead? (also getWorkFlowPriorityMappings ->
getWorkflowPriorityMappings)
* Can we push {{if (overrideWithPriorityMappings) {}} in
WorkflowPriorityMappingsManager#getMappedWorkflowPriorityForApp to the top of
this method? That way there's no impact if this feature's not enabled.
* import java.util.HashSet; is unused in MockRM.java
Looks good other than that though.
> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9760.01.patch, YARN-9760.02.patch
>
>
> Currently priorities are submitted on an application level, but for end users
> it's common to submit workloads to YARN at a workflow level. This jira
> proposes a feature to store workflow id + priority mappings on RM (similar to
> queue mappings). If app is submitted with a certain workflow id (as set in
> application submission context) RM will override this app's priority with the
> one defined in the mapping.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9859) Refactor OpportunisticContainerAllocator
[
https://issues.apache.org/jira/browse/YARN-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939815#comment-16939815
]
Íñigo Goiri commented on YARN-9859:
---
I think the correct indentation would be:
{code}
this.oppContainerAllocator =
new DistributedOpportunisticContainerAllocator(
rmContext.getContainerTokenSecretManager(),
maxAllocationsPerAMHeartbeat);
{code}
Otherwise it is hard to see that they are arguments within the constructor.
> Refactor OpportunisticContainerAllocator
>
>
> Key: YARN-9859
> URL: https://issues.apache.org/jira/browse/YARN-9859
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9859.001.patch, YARN-9859.002.patch
>
>
> Right now OpportunisticContainerAllocator is written mainly for Distributed
> Scheduling and schedules Opportunistic containers on limited set of nodes. As
> part of this jira, we are going to make OpportunisticContainerAllocator as an
> abstract class and DistributedOpportunisticContainerAllocator as actual
> implementation. This would be prerequisite for YARN-9697.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939786#comment-16939786
]
Jonathan Hung commented on YARN-9858:
-
trunk/branch-2 runs pass.
The branch-3.2 failures don't seem related. They fail locally pre-patch for me
too.
The first branch-3.1 test failures was from an out-of-date patch I accidentally
uploaded. The second branch-3.1 run passes.
[~bibinchundatt] mind taking another look? Thanks :)
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-2.001.patch,
> YARN-9858-branch-3.1.001.patch, YARN-9858-branch-3.2.001.patch,
> YARN-9858.001.patch, YARN-9858.002.patch, YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939761#comment-16939761
]
Hadoop QA commented on YARN-9858:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
38s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
47s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
44s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
5s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
40s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 64m
4s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:da675796017 |
| JIRA Issue | YARN-9858 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981593/YARN-9858-branch-2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b9c24b1dbdc2
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939759#comment-16939759
]
Hadoop QA commented on YARN-9858:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
24s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
29s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 69m
48s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:080e9d0f9b3 |
| JIRA Issue | YARN-9858 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981592/YARN-9858-branch-3.1.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 01f319005463 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.1 / 670d065 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24855/testReport/ |
| Max.
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939752#comment-16939752
]
Hadoop QA commented on YARN-9858:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 89m
0s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}173m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | YARN-9858 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981582/YARN-9858.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 821512e0677f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ce58c05 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939751#comment-16939751
]
Hadoop QA commented on YARN-9858:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
12s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
34s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
2s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
49s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 29s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestAppManager |
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:080e9d0f9b3 |
| JIRA Issue | YARN-9858 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981588/YARN-9858-branch-3.1.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a3ca8f610e6a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939748#comment-16939748
]
Hadoop QA commented on YARN-9858:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
22s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
51s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 25s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisherForV2 |
| |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:63396beab41 |
| JIRA Issue | YARN-9858 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981586/YARN-9858-branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 62bfc4d002d8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Updated] (YARN-9864) Format CS Configuration present in Configuration Store
[ https://issues.apache.org/jira/browse/YARN-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9864: Attachment: YARN-9864-001.patch > Format CS Configuration present in Configuration Store > -- > > Key: YARN-9864 > URL: https://issues.apache.org/jira/browse/YARN-9864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9864-001.patch > > > This provides an option to format the configuration changes present in > ConfigurationStore (ZK, LevelDB) and reinitialize from the Local > Capacity-scheduler.xml. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9864) Format CS Configuration present in Configuration Store
[ https://issues.apache.org/jira/browse/YARN-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9864: Component/s: capacity scheduler > Format CS Configuration present in Configuration Store > -- > > Key: YARN-9864 > URL: https://issues.apache.org/jira/browse/YARN-9864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > > This provides an option to format the configuration changes present in > ConfigurationStore (ZK, LevelDB) and reinitialize from the Local > Capacity-scheduler.xml. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9864) Format CS Configuration present in Configuration Store
Prabhu Joseph created YARN-9864: --- Summary: Format CS Configuration present in Configuration Store Key: YARN-9864 URL: https://issues.apache.org/jira/browse/YARN-9864 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 3.3.0 Reporter: Prabhu Joseph Assignee: Prabhu Joseph This provides an option to format the configuration changes present in ConfigurationStore (ZK, LevelDB) and reinitialize from the Local Capacity-scheduler.xml. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939688#comment-16939688
]
Jonathan Hung commented on YARN-9858:
-
* Attached 003 patch which removes the null check as suggested by
[~bibinchundatt].
* Attached branch-3.2 patch which is the same as trunk to trigger jenkins.
* Attached branch-3.1 patch which sets rmcontext's yarn configuration in test
case (same as what was done in YARN-9508)
* Attached branch-2 patch which is the same as branch-3.1 to trigger jenkins.
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-2.001.patch,
> YARN-9858-branch-3.1.001.patch, YARN-9858-branch-3.2.001.patch,
> YARN-9858.001.patch, YARN-9858.002.patch, YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9858:
Attachment: YARN-9858-branch-2.001.patch
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-2.001.patch,
> YARN-9858-branch-3.1.001.patch, YARN-9858-branch-3.2.001.patch,
> YARN-9858.001.patch, YARN-9858.002.patch, YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9858:
Attachment: YARN-9858-branch-3.1.001.patch
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-3.1.001.patch,
> YARN-9858-branch-3.2.001.patch, YARN-9858.001.patch, YARN-9858.002.patch,
> YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9858:
Attachment: (was: YARN-9858-branch-3.1.001.patch)
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-3.2.001.patch, YARN-9858.001.patch,
> YARN-9858.002.patch, YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9858:
Attachment: (was: YARN-9858-branch-2.001.patch)
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-3.2.001.patch, YARN-9858.001.patch,
> YARN-9858.002.patch, YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939680#comment-16939680
]
Hadoop QA commented on YARN-9858:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
2s{color} | {color:red} Docker failed to build yetus/hadoop:da675796017.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9858 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981591/YARN-9858-branch-2.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24854/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-2.001.patch,
> YARN-9858-branch-3.1.001.patch, YARN-9858-branch-3.2.001.patch,
> YARN-9858.001.patch, YARN-9858.002.patch, YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9858:
Attachment: YARN-9858-branch-2.001.patch
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-2.001.patch,
> YARN-9858-branch-3.1.001.patch, YARN-9858-branch-3.2.001.patch,
> YARN-9858.001.patch, YARN-9858.002.patch, YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN
[ https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939677#comment-16939677 ] Eric Badger commented on YARN-9860: --- If there are no log directories, how would you attack debugging container failures? > Enable service mode for Docker containers on YARN > - > > Key: YARN-9860 > URL: https://issues.apache.org/jira/browse/YARN-9860 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9860-001.patch, YARN-9860-002.patch > > > This task is to add support to YARN for running Docker containers in "Service > Mode". > Service Mode - Run the container as defined by the image, but still allow for > injecting configuration. > Background: > Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as > defined in the image. However, still requires modification to official images > due to user propagation > User propagation is problematic for running a secure cluster with sssd > > Implementation: > Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true) > Must be requested at runtime - (example: > YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true) > Entrypoint mode is default enabled for this mode (If Service Mode is > requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set > to true) > Writable log mount will not be added - stdout logging may still work > with entrypoint mode - remove the writable bind mounts > User and groups will not be propagated (now: docker run --user nobody > --group-add=nobody , after: docker run ) > Read-only resources mounted at the file level, files get chmod 777, > parent directory only accessible by the run as user. > cc [~shaneku...@gmail.com] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9858:
Attachment: YARN-9858-branch-3.1.001.patch
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-3.1.001.patch,
> YARN-9858-branch-3.2.001.patch, YARN-9858.001.patch, YARN-9858.002.patch,
> YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9858:
Attachment: (was: YARN-9858-branch-3.1.001.patch)
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-3.2.001.patch, YARN-9858.001.patch,
> YARN-9858.002.patch, YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8403) Nodemanager logs failed to download file with INFO level
[
https://issues.apache.org/jira/browse/YARN-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939660#comment-16939660
]
Brahma Reddy Battula commented on YARN-8403:
IMO,this can be backported to 2.X also can be useful.
> Nodemanager logs failed to download file with INFO level
>
>
> Key: YARN-8403
> URL: https://issues.apache.org/jira/browse/YARN-8403
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8403.001.patch, YARN-8403.002.patch,
> YARN-8403.003.patch, YARN-8403.png
>
>
> Some of the container execution related stack traces are printing in INFO or
> WARN level.
> {code}
> 2018-06-06 03:10:40,077 INFO localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:writeCredentials(1312)) - Writing
> credentials to the nmPrivate file
> /grid/0/hadoop/yarn/local/nmPrivate/container_e02_1528246317583_0048_01_01.tokens
> 2018-06-06 03:10:40,087 INFO localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:run(975)) - Failed to download resource { {
> hdfs://mycluster.example.com:8020/user/hrt_qa/Streaming/InputDir,
> 1528254452720, FILE, null
> },pending,[(container_e02_1528246317583_0048_01_01)],6074418082915225,DOWNLOADING}
> org.apache.hadoop.yarn.exceptions.YarnException: Download and unpack failed
> at
> org.apache.hadoop.yarn.util.FSDownload.downloadAndUnpack(FSDownload.java:306)
> at
> org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:283)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:409)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:66)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.FileNotFoundException:
> /grid/0/hadoop/yarn/local/filecache/28_tmp/InputDir/input1.txt (Permission
> denied)
> at java.io.FileOutputStream.open0(Native Method)
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:236)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:219)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:318)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:307)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:338)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:401)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:408)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:399)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:381)
> at
> org.apache.hadoop.yarn.util.FSDownload.downloadAndUnpack(FSDownload.java:298)
> ... 9 more
> {code}
> {code}
> 2018-06-06 03:10:41,547 WARN privileged.PrivilegedOperationExecutor
> (PrivilegedOperationExecutor.java:executePrivilegedOperation(182)) -
> IOException executing command:
> java.io.InterruptedIOException: java.lang.InterruptedException
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:1012)
> at org.apache.hadoop.util.Shell.run(Shell.java:902)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:152)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:402)
> at
>
[jira] [Updated] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9858:
Attachment: YARN-9858-branch-3.1.001.patch
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-3.1.001.patch,
> YARN-9858-branch-3.2.001.patch, YARN-9858.001.patch, YARN-9858.002.patch,
> YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9858:
Attachment: YARN-9858-branch-3.2.001.patch
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858-branch-3.2.001.patch, YARN-9858.001.patch,
> YARN-9858.002.patch, YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9737) Performance degradation, Distributed Opportunistic Scheduling
[ https://issues.apache.org/jira/browse/YARN-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babble Shack updated YARN-9737: --- Attachment: task_throughput_bp_100j_30t.svg > Performance degradation, Distributed Opportunistic Scheduling > - > > Key: YARN-9737 > URL: https://issues.apache.org/jira/browse/YARN-9737 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 > 1 * Resource Manager – Intel Core i7-4770 CPU @ 3.40GHz, 16GB Memory, 256GB > ssd. > 37 * Node Managers - Intel Core i7-4770 CPU @ 3.40GHz, 8GB Memory, 256GB > ssd. > 2 * 3.5 Gb slots per Node Manager, 1x cpu per slot > yarn-site: [^yarn-site.xml] > yarn-client-yarn-site: [^yarn-client.yarn-site.xml] > >Reporter: Babble Shack >Priority: Major > Labels: performance, scheduler, scheduling > Attachments: jct_100j_30t.png, jct_100j_30t.svg, > jct_cdf_100j_100t_1500.svg, jct_cdf_100j_50t_1500_with_outliers.svg, > jet_boxplot_j100_50t_1500.svg, jet_boxplot_j100_50t_1500_with_outliers.svg, > task_throughput_boxplot_100j_50t_1500.svg, task_throughput_bp_100j_30t.png, > task_throughput_bp_100j_30t.svg, yarn-client.yarn-site.xml, yarn-site.xml > > > Opportunistic scheduling is supposed to provide lower scheduling time, and > thus higher task throughput and lower job completion times for short > jobs/tasks. > Through my experiments I have found distributed scheduling can degrade > performance. > I ran a gridmix trace of 100 short jobs, each with 50 tasks. Average task run > time was 1523ms. > Findings: > * Job completion time, the time take from submitting a job to job > completion, may degrade by over 200% > [^jct_cdf_100j_100t_1500.svg] > [^jct_cdf_100j_50t_1500_with_outliers.svg] > * Job execution time may increase by up to 300% > [^jet_boxplot_j100_50t_1500.svg] > [^jet_boxplot_j100_50t_1500_with_outliers.svg] > * Task throughput decreased by 100% > ^[^task_throughput_boxplot_100j_50t_1500.svg]^ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9737) Performance degradation, Distributed Opportunistic Scheduling
[ https://issues.apache.org/jira/browse/YARN-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babble Shack updated YARN-9737: --- Attachment: jct_100j_30t.svg > Performance degradation, Distributed Opportunistic Scheduling > - > > Key: YARN-9737 > URL: https://issues.apache.org/jira/browse/YARN-9737 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 > 1 * Resource Manager – Intel Core i7-4770 CPU @ 3.40GHz, 16GB Memory, 256GB > ssd. > 37 * Node Managers - Intel Core i7-4770 CPU @ 3.40GHz, 8GB Memory, 256GB > ssd. > 2 * 3.5 Gb slots per Node Manager, 1x cpu per slot > yarn-site: [^yarn-site.xml] > yarn-client-yarn-site: [^yarn-client.yarn-site.xml] > >Reporter: Babble Shack >Priority: Major > Labels: performance, scheduler, scheduling > Attachments: jct_100j_30t.png, jct_100j_30t.svg, > jct_cdf_100j_100t_1500.svg, jct_cdf_100j_50t_1500_with_outliers.svg, > jet_boxplot_j100_50t_1500.svg, jet_boxplot_j100_50t_1500_with_outliers.svg, > task_throughput_boxplot_100j_50t_1500.svg, task_throughput_bp_100j_30t.png, > yarn-client.yarn-site.xml, yarn-site.xml > > > Opportunistic scheduling is supposed to provide lower scheduling time, and > thus higher task throughput and lower job completion times for short > jobs/tasks. > Through my experiments I have found distributed scheduling can degrade > performance. > I ran a gridmix trace of 100 short jobs, each with 50 tasks. Average task run > time was 1523ms. > Findings: > * Job completion time, the time take from submitting a job to job > completion, may degrade by over 200% > [^jct_cdf_100j_100t_1500.svg] > [^jct_cdf_100j_50t_1500_with_outliers.svg] > * Job execution time may increase by up to 300% > [^jet_boxplot_j100_50t_1500.svg] > [^jet_boxplot_j100_50t_1500_with_outliers.svg] > * Task throughput decreased by 100% > ^[^task_throughput_boxplot_100j_50t_1500.svg]^ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9858:
Attachment: YARN-9858.003.patch
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858.001.patch, YARN-9858.002.patch,
> YARN-9858.003.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9737) Performance degradation, Distributed Opportunistic Scheduling
[ https://issues.apache.org/jira/browse/YARN-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babble Shack updated YARN-9737: --- Attachment: jct_100j_30t.png task_throughput_bp_100j_30t.png > Performance degradation, Distributed Opportunistic Scheduling > - > > Key: YARN-9737 > URL: https://issues.apache.org/jira/browse/YARN-9737 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 > 1 * Resource Manager – Intel Core i7-4770 CPU @ 3.40GHz, 16GB Memory, 256GB > ssd. > 37 * Node Managers - Intel Core i7-4770 CPU @ 3.40GHz, 8GB Memory, 256GB > ssd. > 2 * 3.5 Gb slots per Node Manager, 1x cpu per slot > yarn-site: [^yarn-site.xml] > yarn-client-yarn-site: [^yarn-client.yarn-site.xml] > >Reporter: Babble Shack >Priority: Major > Labels: performance, scheduler, scheduling > Attachments: jct_100j_30t.png, jct_cdf_100j_100t_1500.svg, > jct_cdf_100j_50t_1500_with_outliers.svg, jet_boxplot_j100_50t_1500.svg, > jet_boxplot_j100_50t_1500_with_outliers.svg, > task_throughput_boxplot_100j_50t_1500.svg, task_throughput_bp_100j_30t.png, > yarn-client.yarn-site.xml, yarn-site.xml > > > Opportunistic scheduling is supposed to provide lower scheduling time, and > thus higher task throughput and lower job completion times for short > jobs/tasks. > Through my experiments I have found distributed scheduling can degrade > performance. > I ran a gridmix trace of 100 short jobs, each with 50 tasks. Average task run > time was 1523ms. > Findings: > * Job completion time, the time take from submitting a job to job > completion, may degrade by over 200% > [^jct_cdf_100j_100t_1500.svg] > [^jct_cdf_100j_50t_1500_with_outliers.svg] > * Job execution time may increase by up to 300% > [^jet_boxplot_j100_50t_1500.svg] > [^jet_boxplot_j100_50t_1500_with_outliers.svg] > * Task throughput decreased by 100% > ^[^task_throughput_boxplot_100j_50t_1500.svg]^ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Moved] (YARN-9863) Randomize List of Resources to Localize
[
https://issues.apache.org/jira/browse/YARN-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor moved MAPREDUCE-7243 to YARN-9863:
-
Component/s: (was: performance)
(was: nodemanager)
nodemanager
Key: YARN-9863 (was: MAPREDUCE-7243)
Project: Hadoop YARN (was: Hadoop Map/Reduce)
> Randomize List of Resources to Localize
> ---
>
> Key: YARN-9863
> URL: https://issues.apache.org/jira/browse/YARN-9863
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/LocalResourceBuilder.java
> Add a new parameter to {{LocalResourceBuilder}} that allows the list of
> resources to be shuffled randomly. This will allow the Localizer to spread
> the load of requests so that not all of the NodeManagers are requesting to
> localize the same files, in the same order, from the same DataNodes,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9862) yarn-services-core test timeout
[
https://issues.apache.org/jira/browse/YARN-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9862:
Attachment: YARN-9862-001.patch
> yarn-services-core test timeout
> ---
>
> Key: YARN-9862
> URL: https://issues.apache.org/jira/browse/YARN-9862
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9862-001.patch
>
>
> yarn-services-core test timeout. The default timeout of 15minute is not
> sufficient as the TestYarnNativeServices has 14 testcases each with timeout
> of 200 seconds which is not completing within 15minutes.
> {code}
> [WARNING] The requested profile "parallel-tests" could not be activated
> because it does not exist.
> [WARNING] The requested profile "native" could not be activated because it
> does not exist.
> [WARNING] The requested profile "yarn-ui" could not be activated because it
> does not exist.
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test (default-test)
> on project hadoop-yarn-services-core: There was a timeout or other error in
> the fork -> [Help 1]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running
> org.apache.hadoop.yarn.service.timelineservice.TestServiceTimelinePublisher
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.222
> s - in
> org.apache.hadoop.yarn.service.timelineservice.TestServiceTimelinePublisher
> [INFO] Running org.apache.hadoop.yarn.service.monitor.probe.TestDefaultProbe
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.138
> s - in org.apache.hadoop.yarn.service.monitor.probe.TestDefaultProbe
> [INFO] Running org.apache.hadoop.yarn.service.monitor.TestServiceMonitor
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.317
> s - in org.apache.hadoop.yarn.service.monitor.TestServiceMonitor
> [INFO] Running org.apache.hadoop.yarn.service.TestServiceAM
> [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 30.337 s - in org.apache.hadoop.yarn.service.TestServiceAM
> [INFO] Running
> org.apache.hadoop.yarn.service.TestDefaultUpgradeComponentsFinder
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.533
> s - in org.apache.hadoop.yarn.service.TestDefaultUpgradeComponentsFinder
> [INFO] Running org.apache.hadoop.yarn.service.TestServiceManager
> [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.33
> s - in org.apache.hadoop.yarn.service.TestServiceManager
> [INFO] Running org.apache.hadoop.yarn.service.utils.TestFilterUtils
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.799
> s - in org.apache.hadoop.yarn.service.utils.TestFilterUtils
> [INFO] Running org.apache.hadoop.yarn.service.utils.TestServiceApiUtil
> [INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.468
> s - in org.apache.hadoop.yarn.service.utils.TestServiceApiUtil
> [INFO] Running org.apache.hadoop.yarn.service.utils.TestCoreFileSystem
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.699
> s - in org.apache.hadoop.yarn.service.utils.TestCoreFileSystem
> [INFO] Running org.apache.hadoop.yarn.service.provider.TestProviderUtils
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.76 s
> - in org.apache.hadoop.yarn.service.provider.TestProviderUtils
> [INFO] Running
> org.apache.hadoop.yarn.service.provider.TestAbstractProviderService
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.814
> s - in org.apache.hadoop.yarn.service.provider.TestAbstractProviderService
> [INFO] Running
> org.apache.hadoop.yarn.service.component.TestComponentDecommissionInstances
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 108.719 s - in
> org.apache.hadoop.yarn.service.component.TestComponentDecommissionInstances
> [INFO] Running
> org.apache.hadoop.yarn.service.component.TestComponentRestartPolicy
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.108
> s - in org.apache.hadoop.yarn.service.component.TestComponentRestartPolicy
> [INFO] Running org.apache.hadoop.yarn.service.component.TestComponent
> [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.731
> s - in org.apache.hadoop.yarn.service.component.TestComponent
> [INFO] Running
> org.apache.hadoop.yarn.service.component.instance.TestComponentInstance
> [INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.866
> s - in org.apache.hadoop.yarn.service.component.instance.TestComponentInstance
> [INFO] Running
>
[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN
[ https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939534#comment-16939534 ] Prabhu Joseph commented on YARN-9860: - [~skumpf] [~sunilg] Can you review this Jira when you get time. This provides service mode for Docker containers on YARN. The testcase failures are not related and have raised YARN-9862 to fix the same. Thanks. > Enable service mode for Docker containers on YARN > - > > Key: YARN-9860 > URL: https://issues.apache.org/jira/browse/YARN-9860 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9860-001.patch, YARN-9860-002.patch > > > This task is to add support to YARN for running Docker containers in "Service > Mode". > Service Mode - Run the container as defined by the image, but still allow for > injecting configuration. > Background: > Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as > defined in the image. However, still requires modification to official images > due to user propagation > User propagation is problematic for running a secure cluster with sssd > > Implementation: > Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true) > Must be requested at runtime - (example: > YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true) > Entrypoint mode is default enabled for this mode (If Service Mode is > requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set > to true) > Writable log mount will not be added - stdout logging may still work > with entrypoint mode - remove the writable bind mounts > User and groups will not be propagated (now: docker run --user nobody > --group-add=nobody , after: docker run ) > Read-only resources mounted at the file level, files get chmod 777, > parent directory only accessible by the run as user. > cc [~shaneku...@gmail.com] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9860) Enable service mode for Docker containers on YARN
[ https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9860: Attachment: YARN-9860-002.patch > Enable service mode for Docker containers on YARN > - > > Key: YARN-9860 > URL: https://issues.apache.org/jira/browse/YARN-9860 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9860-001.patch, YARN-9860-002.patch > > > This task is to add support to YARN for running Docker containers in "Service > Mode". > Service Mode - Run the container as defined by the image, but still allow for > injecting configuration. > Background: > Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as > defined in the image. However, still requires modification to official images > due to user propagation > User propagation is problematic for running a secure cluster with sssd > > Implementation: > Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true) > Must be requested at runtime - (example: > YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true) > Entrypoint mode is default enabled for this mode (If Service Mode is > requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set > to true) > Writable log mount will not be added - stdout logging may still work > with entrypoint mode - remove the writable bind mounts > User and groups will not be propagated (now: docker run --user nobody > --group-add=nobody , after: docker run ) > Read-only resources mounted at the file level, files get chmod 777, > parent directory only accessible by the run as user. > cc [~shaneku...@gmail.com] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9862) yarn-services-core test timeout
Prabhu Joseph created YARN-9862:
---
Summary: yarn-services-core test timeout
Key: YARN-9862
URL: https://issues.apache.org/jira/browse/YARN-9862
Project: Hadoop YARN
Issue Type: Bug
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph
yarn-services-core test timeout. The default timeout of 15minute is not
sufficient as the TestYarnNativeServices has 14 testcases each with timeout of
200 seconds which is not completing within 15minutes.
{code}
[WARNING] The requested profile "parallel-tests" could not be activated because
it does not exist.
[WARNING] The requested profile "native" could not be activated because it does
not exist.
[WARNING] The requested profile "yarn-ui" could not be activated because it
does not exist.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test (default-test) on
project hadoop-yarn-services-core: There was a timeout or other error in the
fork -> [Help 1]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running
org.apache.hadoop.yarn.service.timelineservice.TestServiceTimelinePublisher
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.222 s
- in org.apache.hadoop.yarn.service.timelineservice.TestServiceTimelinePublisher
[INFO] Running org.apache.hadoop.yarn.service.monitor.probe.TestDefaultProbe
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.138 s
- in org.apache.hadoop.yarn.service.monitor.probe.TestDefaultProbe
[INFO] Running org.apache.hadoop.yarn.service.monitor.TestServiceMonitor
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.317 s
- in org.apache.hadoop.yarn.service.monitor.TestServiceMonitor
[INFO] Running org.apache.hadoop.yarn.service.TestServiceAM
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.337
s - in org.apache.hadoop.yarn.service.TestServiceAM
[INFO] Running org.apache.hadoop.yarn.service.TestDefaultUpgradeComponentsFinder
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.533 s
- in org.apache.hadoop.yarn.service.TestDefaultUpgradeComponentsFinder
[INFO] Running org.apache.hadoop.yarn.service.TestServiceManager
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.33 s
- in org.apache.hadoop.yarn.service.TestServiceManager
[INFO] Running org.apache.hadoop.yarn.service.utils.TestFilterUtils
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.799 s
- in org.apache.hadoop.yarn.service.utils.TestFilterUtils
[INFO] Running org.apache.hadoop.yarn.service.utils.TestServiceApiUtil
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.468 s
- in org.apache.hadoop.yarn.service.utils.TestServiceApiUtil
[INFO] Running org.apache.hadoop.yarn.service.utils.TestCoreFileSystem
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.699 s
- in org.apache.hadoop.yarn.service.utils.TestCoreFileSystem
[INFO] Running org.apache.hadoop.yarn.service.provider.TestProviderUtils
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.76 s -
in org.apache.hadoop.yarn.service.provider.TestProviderUtils
[INFO] Running
org.apache.hadoop.yarn.service.provider.TestAbstractProviderService
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.814 s
- in org.apache.hadoop.yarn.service.provider.TestAbstractProviderService
[INFO] Running
org.apache.hadoop.yarn.service.component.TestComponentDecommissionInstances
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 108.719
s - in
org.apache.hadoop.yarn.service.component.TestComponentDecommissionInstances
[INFO] Running
org.apache.hadoop.yarn.service.component.TestComponentRestartPolicy
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.108 s
- in org.apache.hadoop.yarn.service.component.TestComponentRestartPolicy
[INFO] Running org.apache.hadoop.yarn.service.component.TestComponent
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.731 s
- in org.apache.hadoop.yarn.service.component.TestComponent
[INFO] Running
org.apache.hadoop.yarn.service.component.instance.TestComponentInstance
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.866 s
- in org.apache.hadoop.yarn.service.component.instance.TestComponentInstance
[INFO] Running
org.apache.hadoop.yarn.service.providers.TestAbstractClientProvider
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.447 s
- in org.apache.hadoop.yarn.service.providers.TestAbstractClientProvider
[INFO] Running org.apache.hadoop.yarn.service.providers.TestProviderFactory
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.143 s
- in
[jira] [Commented] (YARN-9859) Refactor OpportunisticContainerAllocator
[
https://issues.apache.org/jira/browse/YARN-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939418#comment-16939418
]
Hadoop QA commented on YARN-9859:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 2 new +
30 unchanged - 2 fixed = 32 total (was 32) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
36s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
16s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m
11s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | YARN-9859 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981516/YARN-9859.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bea3d35cfc2f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939409#comment-16939409
]
Hadoop QA commented on YARN-9858:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
57s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 5s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | YARN-9858 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981521/YARN-9858.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 85720d61c0a3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk /
[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN
[
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939354#comment-16939354
]
Hadoop QA commented on YARN-9860:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 17s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 92 unchanged - 1 fixed = 97 total (was 93) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
28s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 20s{color}
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | YARN-9860 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981534/YARN-9860-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle cc |
| uname | Linux 27658fe680d3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8a9ede5 |
| maven | version: Apache Maven
[jira] [Commented] (YARN-4946) RM should not consider an application as COMPLETED when log aggregation is not in a terminal state
[ https://issues.apache.org/jira/browse/YARN-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939352#comment-16939352 ] Adam Antal commented on YARN-4946: -- I'm +1 (non-binding) on reverting this patch, and check it in detail. I have to add that the state store does not save the Log Aggregation Status, it defaults to "NOT_STARTED". During recovery without final aggregation status the RM probably thinks that those apps haven't been completed (but they did), and keep doing this things. This might have been an edge-case that was not covered in this patch - anyways, I think we should revisit the whole, but as it has severe impact on performance, we should revert it. > RM should not consider an application as COMPLETED when log aggregation is > not in a terminal state > -- > > Key: YARN-4946 > URL: https://issues.apache.org/jira/browse/YARN-4946 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-4946.001.patch, YARN-4946.002.patch, > YARN-4946.003.patch, YARN-4946.004.patch > > > MAPREDUCE-6415 added a tool that combines the aggregated log files for each > Yarn App into a HAR file. When run, it seeds the list by looking at the > aggregated logs directory, and then filters out ineligible apps. One of the > criteria involves checking with the RM that an Application's log aggregation > status is not still running and has not failed. When the RM "forgets" about > an older completed Application (e.g. RM failover, enough time has passed, > etc), the tool won't find the Application in the RM and will just assume that > its log aggregation succeeded, even if it actually failed or is still running. > We can solve this problem by doing the following: > The RM should not consider an app to be fully completed (and thus removed > from its history) until the aggregation status has reached a terminal state > (e.g. SUCCEEDED, FAILED, TIME_OUT). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939325#comment-16939325 ] Peter Bacsko commented on YARN-9699: +1 for standalone tool. Having to have a running RM instance is a hard requirement. > Migration tool that help to generate CS config based on FS config > - > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Gergely Pollak >Priority: Major > Attachments: FS_to_CS_migration_POC.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9861) The ResourceManager log reports an error "Too many open files", the analysis is related to the service
[ https://issues.apache.org/jira/browse/YARN-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939320#comment-16939320 ] Zhankun Tang commented on YARN-9861: [~billie.rinaldi], if any chance, could you please take a look at this? The issue happens when running the submarine per offline discussion. It seems caused by yarn native service leaks the socket/hdfs file handles. Thoughts? > The ResourceManager log reports an error "Too many open files", the analysis > is related to the service > -- > > Key: YARN-9861 > URL: https://issues.apache.org/jira/browse/YARN-9861 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Affects Versions: 3.3.0 > Environment: yarn version:3.3.0-SNAPSHOT > hdfs version:2.7.1 >Reporter: jason >Priority: Major > Attachments: picture1.png, picture2.png, picture3.png, picture4.png, > picture5.png, submarine_kerasgesv2date20190807.json > > > The ResourceManager log outputs "Too many open files" and cannot commit a new > task. > 1. First is the error in picture1, > 2. Then check the file handle open by RM (lsof -p PID), see picture 2, > 3. Also read nameNode audit log (picture 3), > 4. Confirm about service according to the path of service configuration > (picture 4), > 5. Handle number growth trend (picture 5). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9861) The ResourceManager log reports an error "Too many open files", the analysis is related to the service
[ https://issues.apache.org/jira/browse/YARN-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-9861: --- Attachment: submarine_kerasgesv2date20190807.json > The ResourceManager log reports an error "Too many open files", the analysis > is related to the service > -- > > Key: YARN-9861 > URL: https://issues.apache.org/jira/browse/YARN-9861 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Affects Versions: 3.3.0 > Environment: yarn version:3.3.0-SNAPSHOT > hdfs version:2.7.1 >Reporter: jason >Priority: Major > Attachments: picture1.png, picture2.png, picture3.png, picture4.png, > picture5.png, submarine_kerasgesv2date20190807.json > > > The ResourceManager log outputs "Too many open files" and cannot commit a new > task. > 1. First is the error in picture1, > 2. Then check the file handle open by RM (lsof -p PID), see picture 2, > 3. Also read nameNode audit log (picture 3), > 4. Confirm about service according to the path of service configuration > (picture 4), > 5. Handle number growth trend (picture 5). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config
[
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939317#comment-16939317
]
Sunil G commented on YARN-9699:
---
I would vote for standalone like below
{code:java}
Usage: yarn resourcemanager [-format-state-store]
{code}
Above command doesnt need YARN to be running
> Migration tool that help to generate CS config based on FS config
> -
>
> Key: YARN-9699
> URL: https://issues.apache.org/jira/browse/YARN-9699
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wanqiang Ji
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: FS_to_CS_migration_POC.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9861) The ResourceManager log reports an error "Too many open files", the analysis is related to the service
jason created YARN-9861: --- Summary: The ResourceManager log reports an error "Too many open files", the analysis is related to the service Key: YARN-9861 URL: https://issues.apache.org/jira/browse/YARN-9861 Project: Hadoop YARN Issue Type: Bug Components: yarn-native-services Affects Versions: 3.3.0 Environment: yarn version:3.3.0-SNAPSHOT hdfs version:2.7.1 Reporter: jason Attachments: picture1.png, picture2.png, picture3.png, picture4.png, picture5.png The ResourceManager log outputs "Too many open files" and cannot commit a new task. 1. First is the error in picture1, 2. Then check the file handle open by RM (lsof -p PID), see picture 2, 3. Also read nameNode audit log (picture 3), 4. Confirm about service according to the path of service configuration (picture 4), 5. Handle number growth trend (picture 5). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939291#comment-16939291
]
Hadoop QA commented on YARN-9858:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
0s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 30s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | YARN-9858 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981521/YARN-9858.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux aa3f077e9792 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9bf7a6e |
|
[jira] [Updated] (YARN-9860) Enable service mode for Docker containers on YARN
[ https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9860: Attachment: YARN-9860-001.patch > Enable service mode for Docker containers on YARN > - > > Key: YARN-9860 > URL: https://issues.apache.org/jira/browse/YARN-9860 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9860-001.patch > > > This task is to add support to YARN for running Docker containers in "Service > Mode". > Service Mode - Run the container as defined by the image, but still allow for > injecting configuration. > Background: > Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as > defined in the image. However, still requires modification to official images > due to user propagation > User propagation is problematic for running a secure cluster with sssd > > Implementation: > Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true) > Must be requested at runtime - (example: > YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true) > Entrypoint mode is default enabled for this mode (If Service Mode is > requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set > to true) > Writable log mount will not be added - stdout logging may still work > with entrypoint mode - remove the writable bind mounts > User and groups will not be propagated (now: docker run --user nobody > --group-add=nobody , after: docker run ) > Read-only resources mounted at the file level, files get chmod 777, > parent directory only accessible by the run as user. > cc [~shaneku...@gmail.com] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939248#comment-16939248 ] Szilard Nemeth commented on YARN-9699: -- Hi [~sunilg] / [~Prabhu Joseph]! / [~pbacsko]! We have been thinking about the ways we could connect the converter to the existing code so I'm trying to collect all the pros and cons of these approaches. Right now, we can think of a new RM admin CLI command or a completely new, standalone tool to invoke the converter. *RMAdminCLI* Advantages: 1. Classpath is already set so the converter itself is available 2. No need to setup yet another standalone tool 3. yarn-site.xml and fair-scheduler.xml don't need to be specified on CLI as they are given at RM startup. Optionally, these input configs should be Disadvantages: 1. RM needs to be run to be able to convert *Standalone tool* Advantages: 1. Conversion can be started even if RM does not run so offline conversion is available Disadvantages 1. Users need to accustomed to the new standalone tool 2. The new standalone tool should have Yarn RM library on its classpath, this makes things more difficult to set up on clusters Please feel free to add your thoughts as this decision has to be made as early as possible! Thanks! > Migration tool that help to generate CS config based on FS config > - > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Gergely Pollak >Priority: Major > Attachments: FS_to_CS_migration_POC.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939245#comment-16939245
]
Bibin Chundatt commented on YARN-9858:
--
I think we should fix the testcase. Setting conf to rmcontext should solve it ..
{code}
RMContext rmContext = mockRMContext(10, now - 2);
Configuration conf = new YarnConfiguration();
((RMContextImpl)rmContext).setYarnConfiguration(conf);
{code}
Also please a path for branch2 too to trigger jenkins.
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858.001.patch, YARN-9858.002.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5277) when localizers fail due to resource timestamps being out, provide more diagnostics
[ https://issues.apache.org/jira/browse/YARN-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939189#comment-16939189 ] Adam Antal edited comment on YARN-5277 at 9/27/19 7:39 AM: --- Customer of ours is hitting this. If [~ste...@apache.org] is not working on this/does not have a patch, someone else could grab this. was (Author: adam.antal): Customer of ours is hitting this. If [~ste...@apache.org] is not working on this/does not have a patch, someone could grab this. > when localizers fail due to resource timestamps being out, provide more > diagnostics > --- > > Key: YARN-5277 > URL: https://issues.apache.org/jira/browse/YARN-5277 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Major > > When an NM fails a resource D/L as the timestamps are wrong, there's not much > info, just two long values. > It would be good to also include the local time values, *and the current wall > time*. These are the things people need to know when trying to work out what > went wrong -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5277) when localizers fail due to resource timestamps being out, provide more diagnostics
[ https://issues.apache.org/jira/browse/YARN-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939189#comment-16939189 ] Adam Antal commented on YARN-5277: -- Customer of ours is hitting this. If [~ste...@apache.org] is not working on this/does not have a patch, someone could grab this. > when localizers fail due to resource timestamps being out, provide more > diagnostics > --- > > Key: YARN-5277 > URL: https://issues.apache.org/jira/browse/YARN-5277 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Major > > When an NM fails a resource D/L as the timestamps are wrong, there's not much > info, just two long values. > It would be good to also include the local time values, *and the current wall > time*. These are the things people need to know when trying to work out what > went wrong -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939181#comment-16939181
]
Jonathan Hung edited comment on YARN-9858 at 9/27/19 7:29 AM:
--
Thanks [~bibinchundatt]. This won't trigger during normal operation but without
null check , it caused unit tests to fail (see
https://issues.apache.org/jira/browse/YARN-9730?focusedCommentId=16938009=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16938009).
Not sure why jenkins is not running, I triggered a build manually.
[https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/24846]
was (Author: jhung):
Thanks [~bibinchundatt]. This won't trigger during normal operation but without
null check , it caused unit tests to fail (see
https://issues.apache.org/jira/browse/YARN-9730?focusedCommentId=16938009=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16938009).
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858.001.patch, YARN-9858.002.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939181#comment-16939181
]
Jonathan Hung commented on YARN-9858:
-
Thanks [~bibinchundatt]. This won't trigger during normal operation but without
null check , it caused unit tests to fail (see
https://issues.apache.org/jira/browse/YARN-9730?focusedCommentId=16938009=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16938009).
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858.001.patch, YARN-9858.002.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939177#comment-16939177
]
Bibin Chundatt commented on YARN-9858:
--
Over all patch looks good to me.
Minor query .
{code}
3803if (conf == null) {
3804 return new HashSet<>();
3805}
{code}
Check is really required ?
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858.001.patch, YARN-9858.002.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939163#comment-16939163
]
Jonathan Hung commented on YARN-9858:
-
Uploaded 002 patch which implements the above approach. [~bibinchundatt] mind
taking a look? Thanks!
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858.001.patch, YARN-9858.002.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9858:
Attachment: YARN-9858.002.patch
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858.001.patch, YARN-9858.002.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9858) Optimize RMContext getExclusiveEnforcedPartitions
[
https://issues.apache.org/jira/browse/YARN-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939161#comment-16939161
]
Bibin Chundatt commented on YARN-9858:
--
+1 for approach.
> Optimize RMContext getExclusiveEnforcedPartitions
> --
>
> Key: YARN-9858
> URL: https://issues.apache.org/jira/browse/YARN-9858
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Labels: release-blocker
> Attachments: YARN-9858.001.patch
>
>
> Follow-up from YARN-9730. RMContextImpl#getExclusiveEnforcedPartitions is a
> hot code path, need to optimize it .
> Since AMS allocate invoked by multiple handlers locking on conf will occur
> {code}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
> - waiting to lock <0x7f1f8107c748> (a
> org.apache.hadoop.yarn.conf.YarnConfiguration)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
> at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
