date:20190402

[jira] [Commented] (YARN-9394) Use new API of RackResolver to get better performance

2019-04-02 Thread Lantao Jin (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808385#comment-16808385
 ] 

Lantao Jin commented on YARN-9394:
--

Uploaded.

> Use new API of RackResolver to get better performance
> -
>
> Key: YARN-9394
> URL: https://issues.apache.org/jira/browse/YARN-9394
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 3.3.0, 3.2.1
>Reporter: Lantao Jin
>Assignee: Lantao Jin
>Priority: Major
> Attachments: YARN-9394.001.patch, YARN-9394.002.patch
>
>
> After adding a new API in RackResolver YARN-9332, some old callers should 
> switch to new API to get better performance. As an example, Spark 
> [YarnAllocator|https://github.com/apache/spark/blob/733f2c0b98208815f8408e36ab669d7c07e3767f/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L361-L363]
>  for Dynamic Allocation invokes 
> [https://github.com/apache/hadoop/blob/6fa229891e06eea62cb9634efde755f40247e816/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java#L550]
>  to resolve racks in a loop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9394) Use new API of RackResolver to get better performance

2019-04-02 Thread Lantao Jin (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lantao Jin updated YARN-9394:
-
Attachment: YARN-9394.002.patch

> Use new API of RackResolver to get better performance
> -
>
> Key: YARN-9394
> URL: https://issues.apache.org/jira/browse/YARN-9394
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 3.3.0, 3.2.1
>Reporter: Lantao Jin
>Assignee: Lantao Jin
>Priority: Major
> Attachments: YARN-9394.001.patch, YARN-9394.002.patch
>
>
> After adding a new API in RackResolver YARN-9332, some old callers should 
> switch to new API to get better performance. As an example, Spark 
> [YarnAllocator|https://github.com/apache/spark/blob/733f2c0b98208815f8408e36ab669d7c07e3767f/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L361-L363]
>  for Dynamic Allocation invokes 
> [https://github.com/apache/hadoop/blob/6fa229891e06eea62cb9634efde755f40247e816/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java#L550]
>  to resolve racks in a loop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9435) Add Opportunistic Scheduler metrics in ResourceManager.

2019-04-02 Thread Abhishek Modi (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Modi updated YARN-9435:

Attachment: YARN-9435.002.patch

> Add Opportunistic Scheduler metrics in ResourceManager.
> ---
>
> Key: YARN-9435
> URL: https://issues.apache.org/jira/browse/YARN-9435
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9435.001.patch, YARN-9435.002.patch
>
>
> Right now there are no metrics available for Opportunistic Scheduler at 
> ResourceManager. As part of this jira, we will add metrics like number of 
> allocated opportunistic containers, released opportunistic containers, node 
> level allocations, rack level allocations etc. for Opportunistic Scheduler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9396) YARN_RM_CONTAINER_CREATED published twice to ATS

2019-04-02 Thread Prabhu Joseph (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808370#comment-16808370
 ] 

Prabhu Joseph commented on YARN-9396:
-

[~eyang] Can you review this jira when you get time.

> YARN_RM_CONTAINER_CREATED published twice to ATS
> 
>
> Key: YARN-9396
> URL: https://issues.apache.org/jira/browse/YARN-9396
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9396-001.patch
>
>
> RM Container Created event published twice - one from 
> {{ContainerStartedTransition}} (NEW -> ALLOCATED) and another from 
> {{AcquiredTransition}} (ALLOCATED -> ACQUIRED)
> {code}
> 2019-03-18 13:10:13,551 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> container_e11_1552914589043_0001_01_01 Container Transitioned from NEW to 
> ALLOCATED
> 2019-03-18 13:10:13,597 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> container_e11_1552914589043_0001_01_01 Container Transitioned from 
> ALLOCATED to ACQUIRED
> {code}
> *Duplicate Events:*
> {code}
> container_e11_1552914589043_0001_01_01 start:
> 2019-03-18 13:10:13,556 DEBUG 
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
>  Publishing the entity TimelineEntity[type='YARN_CONTAINER', 
> id='container_e11_1552914589043_0001_01_01'], JSON-style content: 
> {"metrics":[],"events":[{"id":"YARN_RM_CONTAINER_CREATED","timestamp":1552914613542,"info":{}}],"createdtime":1552914613542,"idprefix":0,"info":{"YARN_CONTAINER_ALLOCATED_PORT":45454,"YARN_CONTAINER_ALLOCATED_MEMORY":2048,"YARN_CONTAINER_ALLOCATED_PRIORITY":0,"YARN_CONTAINER_ALLOCATED_HOST":"yarn-ats-2","YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":"http://yarn-ats-2:8042","YARN_CONTAINER_ALLOCATED_VCORE":1},"configs":{},"isrelatedto":{},"relatesto":{},"type":"YARN_CONTAINER","id":"container_e11_1552914589043_0001_01_01"}
> 2019-03-18 13:10:13,598 DEBUG 
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
>  Publishing the entity TimelineEntity[type='YARN_CONTAINER', 
> id='container_e11_1552914589043_0001_01_01'], JSON-style content: 
> {"metrics":[],"events":[{"id":"YARN_RM_CONTAINER_CREATED","timestamp":1552914613542,"info":{}}],"createdtime":1552914613542,"idprefix":0,"info":{"YARN_CONTAINER_ALLOCATED_PORT":45454,"YARN_CONTAINER_ALLOCATED_MEMORY":2048,"YARN_CONTAINER_ALLOCATED_PRIORITY":0,"YARN_CONTAINER_ALLOCATED_HOST":"yarn-ats-2","YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":"http://yarn-ats-2:8042","YARN_CONTAINER_ALLOCATED_VCORE":1},"configs":{},"isrelatedto":{},"relatesto":{},"type":"YARN_CONTAINER","id":"container_e11_1552914589043_0001_01_01"}
> container_e11_1552914589043_0001_01_02 start:
> 2019-03-18 13:10:21,599 DEBUG 
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
>  Publishing the entity TimelineEntity[type='YARN_CONTAINER', 
> id='container_e11_1552914589043_0001_01_02'], JSON-style content: 
> {"metrics":[],"events":[{"id":"YARN_RM_CONTAINER_CREATED","timestamp":1552914621596,"info":{}}],"createdtime":1552914621596,"idprefix":0,"info":{"YARN_CONTAINER_ALLOCATED_PORT":45454,"YARN_CONTAINER_ALLOCATED_MEMORY":1024,"YARN_CONTAINER_ALLOCATED_PRIORITY":20,"YARN_CONTAINER_ALLOCATED_HOST":"yarn-ats-2","YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":"http://yarn-ats-2:8042","YARN_CONTAINER_ALLOCATED_VCORE":1},"configs":{},"isrelatedto":{},"relatesto":{},"type":"YARN_CONTAINER","id":"container_e11_1552914589043_0001_01_02"}
> 2019-03-18 13:10:22,344 DEBUG 
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
>  Publishing the entity TimelineEntity[type='YARN_CONTAINER', 
> id='container_e11_1552914589043_0001_01_02'], JSON-style content: 
> {"metrics":[],"events":[{"id":"YARN_RM_CONTAINER_CREATED","timestamp":1552914621596,"info":{}}],"createdtime":1552914621596,"idprefix":0,"info":{"YARN_CONTAINER_ALLOCATED_PORT":45454,"YARN_CONTAINER_ALLOCATED_MEMORY":1024,"YARN_CONTAINER_ALLOCATED_PRIORITY":20,"YARN_CONTAINER_ALLOCATED_HOST":"yarn-ats-2","YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":"http://yarn-ats-2:8042","YARN_CONTAINER_ALLOCATED_VCORE":1},"configs":{},"isrelatedto":{},"relatesto":{},"type":"YARN_CONTAINER","id":"container_e11_1552914589043_0001_01_02"}
> container_e11_1552914589043_0001_01_03 start:
> 2019-03-18 13:10:27,918 DEBUG 
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
>  Publishing the entity TimelineEntity[type='YARN_CONTAINER', 
> id='container_e11_1552914589043_0001_01_03'], JSON-style content: 
> {"metrics":[],"ev

[jira] [Commented] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

2019-04-02 Thread Tarun Parimi (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808365#comment-16808365
 ] 

Tarun Parimi commented on YARN-9336:


The build seems to run in branch-2 properly for the same patch. [~rohithsharma] 
, please review and commit to branch-2 when you get time.

> JobHistoryServer leaks CLOSE_WAIT tcp connections when using 
> LogAggregationIndexedFileController
> 
>
> Key: YARN-9336
> URL: https://issues.apache.org/jira/browse/YARN-9336
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9336-branch-2.001.patch, 
> YARN-9336-branch-2.9.001.patch, YARN-9336-branch-2.9.002.patch, 
> YARN-9336-branch-2.9.003.patch, YARN-9336.001.patch, YARN-9336.002.patch
>
>
> The JobHistoryServer is leaking CLOSE_WAIT connections to DataNodes whenever 
> viewing a huge log file in JobhistoryServer. This happens only when the below 
> is configured.
> {code:java}
> yarn.log-aggregation.file-formats=IndexedFormat
> yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
> yarn.log.server.url=http://jobhistory-host:19888/jobhistory/logs
> {code}
> On investigation, I found that the FSDataInputStream is not closed in 
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.IndexedFileAggregatedLogsBlock
>  . Since this block is called every time the Jobhistory page displays the 
> logs, CLOSE_WAIT connections to DataNodes keep on increasing in 
> JobHistoryServer.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support

2019-04-02 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808125#comment-16808125
 ] 

Hadoop QA commented on YARN-9123:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 16s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 24s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
 The patch generated 1 new + 5 unchanged - 1 fixed = 6 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 40s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m 
57s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m  6s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9123 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12958729/YARN-9123.006.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 2041b4ff4bee 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bfc90bd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/23867/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/23867/testReport/ |
| Max. process+thread count | 419 (vs. ulimit of 1) |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U

[jira] [Commented] (YARN-9435) Add Opportunistic Scheduler metrics in ResourceManager.

2019-04-02 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808096#comment-16808096
 ] 

Hadoop QA commented on YARN-9435:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
 2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 30s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 54s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 8 new + 
36 unchanged - 0 fixed = 44 total (was 36) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 30s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
44s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m  8s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 16s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService 
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9435 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12964598/YARN-9435.001.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 25c81952f333 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | t

[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support

2019-04-02 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808093#comment-16808093
 ] 

Szilard Nemeth commented on YARN-9123:
--

Thanks [~jojochuang] for taking a look at this!
If it turns put I need to fix the checkstyle warnings, I will fix them 
tomorrow. 

> Clean up and split testcases in TestNMWebServices for GPU support
> -
>
> Key: YARN-9123
> URL: https://issues.apache.org/jira/browse/YARN-9123
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-9123.001.patch, YARN-9123.002.patch, 
> YARN-9123.003.patch, YARN-9123.004.patch, YARN-9123.005.patch, 
> YARN-9123.006.patch
>
>
> The following testcases can be cleaned up a bit: 
> TestNMWebServices#testGetNMResourceInfo - Can be split up to 3 different cases
> TestNMWebServices#testGetYarnGpuResourceInfo



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9281) Add express upgrade button to Appcatalog UI

2019-04-02 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808061#comment-16808061
 ] 

Hadoop QA commented on YARN-9281:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 52s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m  6s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
51s{color} | {color:green} hadoop-yarn-applications-catalog-webapp in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 12s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9281 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12964605/YARN-9281.007.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  xml  findbugs  checkstyle  |
| uname | Linux ae9548d35246 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bfc90bd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/23866/testReport/ |
| Max. process+thread count | 415 (vs. ulimit of 1) |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-ca

[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support

2019-04-02 Thread Wei-Chiu Chuang (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808056#comment-16808056
 ] 

Wei-Chiu Chuang commented on YARN-9123:
---

Looks good to me. I retriggered the precommit check to understand where the 
checkstyle warning comes from.

> Clean up and split testcases in TestNMWebServices for GPU support
> -
>
> Key: YARN-9123
> URL: https://issues.apache.org/jira/browse/YARN-9123
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-9123.001.patch, YARN-9123.002.patch, 
> YARN-9123.003.patch, YARN-9123.004.patch, YARN-9123.005.patch, 
> YARN-9123.006.patch
>
>
> The following testcases can be cleaned up a bit: 
> TestNMWebServices#testGetNMResourceInfo - Can be split up to 3 different cases
> TestNMWebServices#testGetYarnGpuResourceInfo



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.

2019-04-02 Thread JIRA



[ 
https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808018#comment-16808018
 ] 

Íñigo Goiri commented on YARN-999:
--

Anybody available for review?
[~djp]?

> In case of long running tasks, reduce node resource should balloon out 
> resource quickly by calling preemption API and suspending running task. 
> ---
>
> Key: YARN-999
> URL: https://issues.apache.org/jira/browse/YARN-999
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: graceful, nodemanager, scheduler
>Reporter: Junping Du
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: YARN-291.000.patch, YARN-999.001.patch, 
> YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch, 
> YARN-999.005.patch, YARN-999.006.patch, YARN-999.007.patch, 
> YARN-999.008.patch, YARN-999.009.patch
>
>
> In current design and implementation, when we decrease resource on node to 
> less than resource consumption of current running tasks, tasks can still be 
> running until the end. But just no new task get assigned on this node 
> (because AvailableResource < 0) until some tasks are finished and 
> AvailableResource > 0 again. This is good for most cases but in case of long 
> running task, it could be too slow for resource setting to actually work so 
> preemption could be used here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9281) Add express upgrade button to Appcatalog UI

2019-04-02 Thread Eric Yang (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807993#comment-16807993
 ] 

Eric Yang commented on YARN-9281:
-

Patch 7 fixes the code style issues identified by [~adam.antal].

> Add express upgrade button to Appcatalog UI
> ---
>
> Key: YARN-9281
> URL: https://issues.apache.org/jira/browse/YARN-9281
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9281.001.patch, YARN-9281.002.patch, 
> YARN-9281.003.patch, YARN-9281.004.patch, YARN-9281.005.patch, 
> YARN-9281.006.patch, YARN-9281.007.patch
>
>
> It would be nice to have ability to upgrade applications deployed by 
> Application catalog from Application catalog UI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9281) Add express upgrade button to Appcatalog UI

2019-04-02 Thread Eric Yang (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated YARN-9281:

Attachment: YARN-9281.007.patch

> Add express upgrade button to Appcatalog UI
> ---
>
> Key: YARN-9281
> URL: https://issues.apache.org/jira/browse/YARN-9281
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9281.001.patch, YARN-9281.002.patch, 
> YARN-9281.003.patch, YARN-9281.004.patch, YARN-9281.005.patch, 
> YARN-9281.006.patch, YARN-9281.007.patch
>
>
> It would be nice to have ability to upgrade applications deployed by 
> Application catalog from Application catalog UI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9281) Add express upgrade button to Appcatalog UI

2019-04-02 Thread Eric Yang (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807952#comment-16807952
 ] 

Eric Yang commented on YARN-9281:
-

{quote}I think it would be neat, if it was moved to a separate function 
(something like private void commitSolrChanges()).{quote}

Sounds good, will update.

{quote}We iterate over the appEntries and only the check the last element. Is 
it intended? If yes, then use appEntries.get(appEntries.size() - 1), if no, 
then move the assert into the for cycle to check every element.{quote}

The test case only has one application.  For loop isn't really necessary, and I 
will update to use appEntries.get(appEntries.size() -1) instead.  Thanks

> Add express upgrade button to Appcatalog UI
> ---
>
> Key: YARN-9281
> URL: https://issues.apache.org/jira/browse/YARN-9281
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9281.001.patch, YARN-9281.002.patch, 
> YARN-9281.003.patch, YARN-9281.004.patch, YARN-9281.005.patch, 
> YARN-9281.006.patch
>
>
> It would be nice to have ability to upgrade applications deployed by 
> Application catalog from Application catalog UI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-9435) Add Opportunistic Scheduler metrics in ResourceManager.

2019-04-02 Thread Abhishek Modi (JIRA)

Abhishek Modi created YARN-9435:
---

 Summary: Add Opportunistic Scheduler metrics in ResourceManager.
 Key: YARN-9435
 URL: https://issues.apache.org/jira/browse/YARN-9435
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Abhishek Modi
Assignee: Abhishek Modi


Right now there are no metrics available for Opportunistic Scheduler at 
ResourceManager. As part of this jira, we will add metrics like number of 
allocated opportunistic containers, released opportunistic containers, node 
level allocations, rack level allocations etc. for Opportunistic Scheduler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9424) Change getDeclaredMethods to getMethods in FederationClientInterceptor#invokeConcurrent()

2019-04-02 Thread Bibin A Chundatt (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bibin A Chundatt updated YARN-9424:
---
Component/s: federation

> Change getDeclaredMethods to getMethods in 
> FederationClientInterceptor#invokeConcurrent()
> -
>
> Key: YARN-9424
> URL: https://issues.apache.org/jira/browse/YARN-9424
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: federation
>Reporter: Shen Yinjie
>Assignee: Shen Yinjie
>Priority: Major
> Attachments: YARN-9124_1.patch
>
>
> In YARN-8699, FederationClientInterceptor#invokeConcurrent uses 
> getDeclaredMethods(), which cannot recongnize some methods in 
> ApplicationBaseProtocol (ApplicationClientProtocol extend 
> ApplicationBaseProtocol) .
> We have implemented some methods in FederationClientInterceptor, such as 
> getApplications(), GetQueueUserAclsInfo ...etc, when I run "yarn application 
> -list" by connecting to yarn router, router will throw exception.
> So change getDeclaredMethods() to getMethods().



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4901) MockRM should clear the QueueMetrics when it starts

2019-04-02 Thread Sunil Govindan (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807770#comment-16807770
 ] 

Sunil Govindan commented on YARN-4901:
--

+1. Lets get this in.

> MockRM should clear the QueueMetrics when it starts
> ---
>
> Key: YARN-4901
> URL: https://issues.apache.org/jira/browse/YARN-4901
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Daniel Templeton
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-4901-001.patch
>
>
> The {{ResourceManager}} rightly assumes that when it starts, it's starting 
> from naught.  The {{MockRM}}, however, violates that assumption.  For 
> example, in {{TestNMReconnect}}, each test method creates a new {{MockRM}} 
> instance.  The {{QueueMetrics.queueMetrics}} field is static, which means 
> that when multiple {{MockRM}} instances are created, the {{QueueMetrics}} 
> bleed over.  Having the MockRM clear the {{QueueMetrics}} when it starts 
> should resolve the issue.  I haven't looked yet at scope to see how hard easy 
> that is to do.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

2019-04-02 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807659#comment-16807659
 ] 

Hadoop QA commented on YARN-9336:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
49s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} branch-2 passed with JDK v1.8.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
44s{color} | {color:green} branch-2 passed with JDK v1.8.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed with JDK v1.8.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 22s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 3 new + 
1 unchanged - 1 fixed = 4 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed with JDK v1.8.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
53s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m  5s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:da67579 |
| JIRA Issue | YARN-9336 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12964545/YARN-9336-branch-2.001.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 82cab6540fa1 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision |

[jira] [Updated] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

2019-04-02 Thread Tarun Parimi (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tarun Parimi updated YARN-9336:
---
Attachment: YARN-9336-branch-2.001.patch

> JobHistoryServer leaks CLOSE_WAIT tcp connections when using 
> LogAggregationIndexedFileController
> 
>
> Key: YARN-9336
> URL: https://issues.apache.org/jira/browse/YARN-9336
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9336-branch-2.001.patch, 
> YARN-9336-branch-2.9.001.patch, YARN-9336-branch-2.9.002.patch, 
> YARN-9336-branch-2.9.003.patch, YARN-9336.001.patch, YARN-9336.002.patch
>
>
> The JobHistoryServer is leaking CLOSE_WAIT connections to DataNodes whenever 
> viewing a huge log file in JobhistoryServer. This happens only when the below 
> is configured.
> {code:java}
> yarn.log-aggregation.file-formats=IndexedFormat
> yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
> yarn.log.server.url=http://jobhistory-host:19888/jobhistory/logs
> {code}
> On investigation, I found that the FSDataInputStream is not closed in 
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.IndexedFileAggregatedLogsBlock
>  . Since this block is called every time the Jobhistory page displays the 
> logs, CLOSE_WAIT connections to DataNodes keep on increasing in 
> JobHistoryServer.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9432) Excess reserved containers may exist for a long time after its request has been cancelled or satisfied when multi-nodes enabled

2019-04-02 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807613#comment-16807613
 ] 

Hadoop QA commented on YARN-9432:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 34s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 58s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 53s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 10s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9432 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12964534/YARN-9432.002.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux b0fda0ff5ba2 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2f75283 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/23862/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/23862/testReport/ |
| Max. process+thread count | 881 (vs. ulimit of 1) |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hado

[jira] [Commented] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

2019-04-02 Thread Tarun Parimi (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807607#comment-16807607
 ] 

Tarun Parimi commented on YARN-9336:


Hi [~rohithsharma] ,

I have attached a branch-2.9 patch for backport . Looks like a jenkins issue in 
branch-2.9 is causing the build to fail. Local build for hadoop branch-2.9 runs 
fine with this patch. I have also manually verified the fix in my local 
cluster. The patch contains the same change as the one you already committed in 
trunk. Please review and commit to branch-2.9 when you get time.

> JobHistoryServer leaks CLOSE_WAIT tcp connections when using 
> LogAggregationIndexedFileController
> 
>
> Key: YARN-9336
> URL: https://issues.apache.org/jira/browse/YARN-9336
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9336-branch-2.9.001.patch, 
> YARN-9336-branch-2.9.002.patch, YARN-9336-branch-2.9.003.patch, 
> YARN-9336.001.patch, YARN-9336.002.patch
>
>
> The JobHistoryServer is leaking CLOSE_WAIT connections to DataNodes whenever 
> viewing a huge log file in JobhistoryServer. This happens only when the below 
> is configured.
> {code:java}
> yarn.log-aggregation.file-formats=IndexedFormat
> yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
> yarn.log.server.url=http://jobhistory-host:19888/jobhistory/logs
> {code}
> On investigation, I found that the FSDataInputStream is not closed in 
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.IndexedFileAggregatedLogsBlock
>  . Since this block is called every time the Jobhistory page displays the 
> logs, CLOSE_WAIT connections to DataNodes keep on increasing in 
> JobHistoryServer.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9326) Fair Scheduler configuration defaults are not documented in case of min and maxResources

2019-04-02 Thread Adam Antal (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807605#comment-16807605
 ] 

Adam Antal commented on YARN-9326:
--

[~templedf], do you have any further comments?

> Fair Scheduler configuration defaults are not documented in case of min and 
> maxResources
> 
>
> Key: YARN-9326
> URL: https://issues.apache.org/jira/browse/YARN-9326
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: docs, documentation, fairscheduler, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-9326.001.patch, YARN-9326.002.patch, 
> YARN-9326.003.patch, YARN-9326.004.patch, YARN-9326.005.patch
>
>
> The FairScheduler's configuration has the following defaults (from the code: 
> javadoc):
> {noformat}
> In new style resources, any resource that is not specified will be set to 
> missing or 0%, as appropriate. Also, in the new style resources, units are 
> not allowed. Units are assumed from the resource manager's settings for the 
> resources when the value isn't a percentage. The missing parameter is only 
> used in the case of new style resources without percentages. With new style 
> resources with percentages, any missing resources will be assumed to be 100% 
> because percentages are only used with maximum resource limits.
> {noformat}
> This is not documented in the hadoop yarn site FairScheduler.html. It is 
> quite intuitive, but still need to be documented though.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

2019-04-02 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807597#comment-16807597
 ] 

Hadoop QA commented on YARN-9336:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.9 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m  
9s{color} | {color:red} root in branch-2.9 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m  
8s{color} | {color:red} hadoop-yarn-common in branch-2.9 failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m  8s{color} | {color:orange} The patch fails to run checkstyle in 
hadoop-yarn-common {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m  
8s{color} | {color:red} hadoop-yarn-common in branch-2.9 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m  
8s{color} | {color:red} hadoop-yarn-common in branch-2.9 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m  
8s{color} | {color:red} hadoop-yarn-common in branch-2.9 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m  
8s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m  
8s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m  7s{color} 
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m  6s{color} | {color:orange} The patch fails to run checkstyle in 
hadoop-yarn-common {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m  
7s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m  
8s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m  
8s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m  7s{color} 
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:blue}0{color} | {color:blue} asflicense {color} | {color:blue}  0m  
9s{color} | {color:blue} ASF License check generated no output? {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 51s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:07598f5 |
| JIRA Issue | YARN-9336 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12964540/YARN-9336-branch-2.9.003.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 3b5099397ecf 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.9 / c7a60ca |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.7.0_95 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-YARN-Build/23863/artifact/out/branch-mvninstall-root.txt
 |
| compile | 
https://builds.apache.org/job/PreCommit-YARN-Build/23863/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/23863/artifact/out//testptch/patchprocess/maven-branch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
 |
| mvnsite | 
https://builds.apache.org/job/PreCommit-YARN-Build/23863/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-co

[jira] [Updated] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

2019-04-02 Thread Tarun Parimi (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tarun Parimi updated YARN-9336:
---
Attachment: YARN-9336-branch-2.9.003.patch

> JobHistoryServer leaks CLOSE_WAIT tcp connections when using 
> LogAggregationIndexedFileController
> 
>
> Key: YARN-9336
> URL: https://issues.apache.org/jira/browse/YARN-9336
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9336-branch-2.9.001.patch, 
> YARN-9336-branch-2.9.002.patch, YARN-9336-branch-2.9.003.patch, 
> YARN-9336.001.patch, YARN-9336.002.patch
>
>
> The JobHistoryServer is leaking CLOSE_WAIT connections to DataNodes whenever 
> viewing a huge log file in JobhistoryServer. This happens only when the below 
> is configured.
> {code:java}
> yarn.log-aggregation.file-formats=IndexedFormat
> yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
> yarn.log.server.url=http://jobhistory-host:19888/jobhistory/logs
> {code}
> On investigation, I found that the FSDataInputStream is not closed in 
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.IndexedFileAggregatedLogsBlock
>  . Since this block is called every time the Jobhistory page displays the 
> logs, CLOSE_WAIT connections to DataNodes keep on increasing in 
> JobHistoryServer.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4901) MockRM should clear the QueueMetrics when it starts

2019-04-02 Thread Peter Bacsko (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807557#comment-16807557
 ] 

Peter Bacsko commented on YARN-4901:


[~sunilg] can you review & commit this patch? 

> MockRM should clear the QueueMetrics when it starts
> ---
>
> Key: YARN-4901
> URL: https://issues.apache.org/jira/browse/YARN-4901
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Daniel Templeton
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-4901-001.patch
>
>
> The {{ResourceManager}} rightly assumes that when it starts, it's starting 
> from naught.  The {{MockRM}}, however, violates that assumption.  For 
> example, in {{TestNMReconnect}}, each test method creates a new {{MockRM}} 
> instance.  The {{QueueMetrics.queueMetrics}} field is static, which means 
> that when multiple {{MockRM}} instances are created, the {{QueueMetrics}} 
> bleed over.  Having the MockRM clear the {{QueueMetrics}} when it starts 
> should resolve the issue.  I haven't looked yet at scope to see how hard easy 
> that is to do.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9281) Add express upgrade button to Appcatalog UI

2019-04-02 Thread Adam Antal (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807527#comment-16807527
 ] 

Adam Antal commented on YARN-9281:
--

Hi [~eyang],

I hope you don't mind that I took a look at the patch. I'm not much into js, so 
I focused on the java part. 
Overall looks great (I'm still learning the basics though, so forgive me).

This piece of code is replicated multiple times in 
{{AppCatalogSolrClient.java}}. I think it would be neat, if it was moved to a 
separate function (something like private void commitSolrChanges()).
{noformat}
// Commit Solr changes.
UpdateResponse detailsResponse = solr.add(docs);
if (detailsResponse.getStatus() != 0) {
   throw new IOException("Unable to register docker instance "
   + "with application entry.");
}
solr.commit();
{noformat}

I don't really understand the end of the 
{{TestAppCatalogSolrClient$testUpgradeApp}} testcase:
{noformat}
List appEntries = spy.listAppEntries();
for (AppEntry entry : appEntries) {
   Service tmp = entry.getYarnfile();
   actual = tmp.getVersion();
}
assertEquals(expected, actual);
{noformat}
We iterate over the appEntries and only the check the last element. Is it 
intended? If yes, then use appEntries.get(appEntries.size() - 1), if no, then 
move the assert into the for cycle to check every element.

Also you can modify the interior of that cycle to 
{noformat}
String tmp = entry.getYarnfile().getVersion();
{noformat}
(and the Service import is not needed anymore in 
{{TestAppCatalogSolrClient.java}}).

> Add express upgrade button to Appcatalog UI
> ---
>
> Key: YARN-9281
> URL: https://issues.apache.org/jira/browse/YARN-9281
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9281.001.patch, YARN-9281.002.patch, 
> YARN-9281.003.patch, YARN-9281.004.patch, YARN-9281.005.patch, 
> YARN-9281.006.patch
>
>
> It would be nice to have ability to upgrade applications deployed by 
> Application catalog from Application catalog UI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9434) Evict the container that caused NM host high load

2019-04-02 Thread liyakun (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyakun updated YARN-9434:
--
Description: Evict the container that caused NM host high load to reduce 
the impact on other containers.  (was: Evict the container that caused NM host 
high load to )

> Evict the container that caused NM host high load
> -
>
> Key: YARN-9434
> URL: https://issues.apache.org/jira/browse/YARN-9434
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Reporter: liyakun
>Assignee: liyakun
>Priority: Major
>
> Evict the container that caused NM host high load to reduce the impact on 
> other containers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9434) Evict the container that caused NM host high load

2019-04-02 Thread liyakun (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyakun updated YARN-9434:
--
Description: Evict the container that caused NM host high load to 

> Evict the container that caused NM host high load
> -
>
> Key: YARN-9434
> URL: https://issues.apache.org/jira/browse/YARN-9434
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Reporter: liyakun
>Assignee: liyakun
>Priority: Major
>
> Evict the container that caused NM host high load to 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9345) NM actively does not accept new containers in the heartbeat

2019-04-02 Thread liyakun (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyakun updated YARN-9345:
--
Issue Type: Improvement  (was: New Feature)

> NM actively does not accept new containers in the heartbeat
> ---
>
> Key: YARN-9345
> URL: https://issues.apache.org/jira/browse/YARN-9345
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Reporter: liyakun
>Assignee: liyakun
>Priority: Major
>
> At present, NM has only one health check mechanism. If it enters an unhealthy 
> state, all the containers running on it will be killed.
>  However, the unhealthy condition of node can be divided into two types, one 
> is long-term unavailable (current health mechanism), and the other is only 
> temporary pressure.
>  For temporary stress, node only needs to wait for a while to return to 
> normal (such as temporary load high).
>  To do this, we need to extend the functionality of the health check to join 
> the state of temporarily not accepting new tasks(do not kill the container 
> that is already running).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9434) Evict the container that caused NM host high load

2019-04-02 Thread liyakun (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyakun updated YARN-9434:
--
Summary: Evict the container that caused NM host high load  (was: evict the 
container that caused NM host high load)

> Evict the container that caused NM host high load
> -
>
> Key: YARN-9434
> URL: https://issues.apache.org/jira/browse/YARN-9434
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Reporter: liyakun
>Assignee: liyakun
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-9434) evict the container that caused NM host high load

2019-04-02 Thread liyakun (JIRA)

liyakun created YARN-9434:
-

 Summary: evict the container that caused NM host high load
 Key: YARN-9434
 URL: https://issues.apache.org/jira/browse/YARN-9434
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager
Reporter: liyakun
Assignee: liyakun






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9429) A status code error in ResourceManager REST api doc

2019-04-02 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807479#comment-16807479
 ] 

Szilard Nemeth commented on YARN-9429:
--

Hi [~lingjinjiang]!

Thanks for fixing this, +1 (non-binding)

> A status code error in ResourceManager REST api doc
> ---
>
> Key: YARN-9429
> URL: https://issues.apache.org/jira/browse/YARN-9429
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Reporter: Jinjiang Ling
>Assignee: Jinjiang Ling
>Priority: Major
> Attachments: YARN-9429.001.patch
>
>
> A status code error in ResourceManager api docs.
> In section "Cluster Application State API",the unauthorized error response 
> header is described below.
> {code:java}
> Response Header:
> HTTP/1.1 403 Unauthorized
> Server: Jetty(6.1.26)
> {code}
> As commonly known, the unauthorized status code should be *401*.
>   
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9432) Excess reserved containers may exist for a long time after its request has been cancelled or satisfied when multi-nodes enabled

2019-04-02 Thread Tao Yang (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Yang updated YARN-9432:
---
Attachment: YARN-9432.002.patch

> Excess reserved containers may exist for a long time after its request has 
> been cancelled or satisfied when multi-nodes enabled
> ---
>
> Key: YARN-9432
> URL: https://issues.apache.org/jira/browse/YARN-9432
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacityscheduler
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-9432.001.patch, YARN-9432.002.patch
>
>
> Reserved containers may change to be excess after its request has been 
> cancelled or satisfied, excess reserved containers need to be unreserved 
> quickly to release resource for others.
> For multi-nodes disabled scenario, excess reserved containers can be quickly 
> released in next node heartbeat, the calling stack is 
> CapacityScheduler#nodeUpdate -->  CapacityScheduler#allocateContainersToNode 
> --> CapacityScheduler#allocateContainerOnSingleNode. 
> But for multi-nodes enabled scenario, excess reserved containers have chance 
> to be released only in allocation process, key phase of the calling stack is 
> LeafQueue#assignContainers --> LeafQueue#allocateFromReservedContainer. 
> According to this, excess reserved containers may not be released until its 
> queue has pending request and has chance to be allocated, and the worst is 
> that excess reserved containers will never be released and keep holding 
> resource if there is no additional pending request for this queue.
> To solve this problem, my opinion is to directly kill excess reserved 
> containers when request is satisfied (in FiCaSchedulerApp#apply) or the 
> allocation number of resource-requests/scheduling-requests is updated to be 0 
> (in SchedulerApplicationAttempt#updateResourceRequests / 
> SchedulerApplicationAttempt#updateSchedulingRequests).
> Please feel free to give your suggestions. Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9432) Excess reserved containers may exist for a long time after its request has been cancelled or satisfied when multi-nodes enabled

2019-04-02 Thread Tao Yang (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807475#comment-16807475
 ] 

Tao Yang commented on YARN-9432:


Attached v2 patch to fix check-style error and UT failures.

> Excess reserved containers may exist for a long time after its request has 
> been cancelled or satisfied when multi-nodes enabled
> ---
>
> Key: YARN-9432
> URL: https://issues.apache.org/jira/browse/YARN-9432
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacityscheduler
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-9432.001.patch, YARN-9432.002.patch
>
>
> Reserved containers may change to be excess after its request has been 
> cancelled or satisfied, excess reserved containers need to be unreserved 
> quickly to release resource for others.
> For multi-nodes disabled scenario, excess reserved containers can be quickly 
> released in next node heartbeat, the calling stack is 
> CapacityScheduler#nodeUpdate -->  CapacityScheduler#allocateContainersToNode 
> --> CapacityScheduler#allocateContainerOnSingleNode. 
> But for multi-nodes enabled scenario, excess reserved containers have chance 
> to be released only in allocation process, key phase of the calling stack is 
> LeafQueue#assignContainers --> LeafQueue#allocateFromReservedContainer. 
> According to this, excess reserved containers may not be released until its 
> queue has pending request and has chance to be allocated, and the worst is 
> that excess reserved containers will never be released and keep holding 
> resource if there is no additional pending request for this queue.
> To solve this problem, my opinion is to directly kill excess reserved 
> containers when request is satisfied (in FiCaSchedulerApp#apply) or the 
> allocation number of resource-requests/scheduling-requests is updated to be 0 
> (in SchedulerApplicationAttempt#updateResourceRequests / 
> SchedulerApplicationAttempt#updateSchedulingRequests).
> Please feel free to give your suggestions. Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9424) Change getDeclaredMethods to getMethods in FederationClientInterceptor#invokeConcurrent()

2019-04-02 Thread Shen Yinjie (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807467#comment-16807467
 ] 

Shen Yinjie commented on YARN-9424:
---

Thanks [~giovanni.fumarola] for comment!

> Change getDeclaredMethods to getMethods in 
> FederationClientInterceptor#invokeConcurrent()
> -
>
> Key: YARN-9424
> URL: https://issues.apache.org/jira/browse/YARN-9424
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Shen Yinjie
>Assignee: Shen Yinjie
>Priority: Major
> Attachments: YARN-9124_1.patch
>
>
> In YARN-8699, FederationClientInterceptor#invokeConcurrent uses 
> getDeclaredMethods(), which cannot recongnize some methods in 
> ApplicationBaseProtocol (ApplicationClientProtocol extend 
> ApplicationBaseProtocol) .
> We have implemented some methods in FederationClientInterceptor, such as 
> getApplications(), GetQueueUserAclsInfo ...etc, when I run "yarn application 
> -list" by connecting to yarn router, router will throw exception.
> So change getDeclaredMethods() to getMethods().



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9394) Use new API of RackResolver to get better performance

[jira] [Updated] (YARN-9394) Use new API of RackResolver to get better performance

[jira] [Updated] (YARN-9435) Add Opportunistic Scheduler metrics in ResourceManager.

[jira] [Commented] (YARN-9396) YARN_RM_CONTAINER_CREATED published twice to ATS

[jira] [Commented] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support

[jira] [Commented] (YARN-9435) Add Opportunistic Scheduler metrics in ResourceManager.

[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support

[jira] [Commented] (YARN-9281) Add express upgrade button to Appcatalog UI

[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support

[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.

[jira] [Commented] (YARN-9281) Add express upgrade button to Appcatalog UI

[jira] [Updated] (YARN-9281) Add express upgrade button to Appcatalog UI

[jira] [Commented] (YARN-9281) Add express upgrade button to Appcatalog UI

[jira] [Created] (YARN-9435) Add Opportunistic Scheduler metrics in ResourceManager.

[jira] [Updated] (YARN-9424) Change getDeclaredMethods to getMethods in FederationClientInterceptor#invokeConcurrent()

[jira] [Commented] (YARN-4901) MockRM should clear the QueueMetrics when it starts

[jira] [Commented] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

[jira] [Updated] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

[jira] [Commented] (YARN-9432) Excess reserved containers may exist for a long time after its request has been cancelled or satisfied when multi-nodes enabled

[jira] [Commented] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

[jira] [Commented] (YARN-9326) Fair Scheduler configuration defaults are not documented in case of min and maxResources

[jira] [Commented] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

[jira] [Updated] (YARN-9336) JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

[jira] [Commented] (YARN-4901) MockRM should clear the QueueMetrics when it starts

[jira] [Commented] (YARN-9281) Add express upgrade button to Appcatalog UI

[jira] [Updated] (YARN-9434) Evict the container that caused NM host high load

[jira] [Updated] (YARN-9434) Evict the container that caused NM host high load

[jira] [Updated] (YARN-9345) NM actively does not accept new containers in the heartbeat

[jira] [Updated] (YARN-9434) Evict the container that caused NM host high load

[jira] [Created] (YARN-9434) evict the container that caused NM host high load

[jira] [Commented] (YARN-9429) A status code error in ResourceManager REST api doc

[jira] [Updated] (YARN-9432) Excess reserved containers may exist for a long time after its request has been cancelled or satisfied when multi-nodes enabled

[jira] [Commented] (YARN-9432) Excess reserved containers may exist for a long time after its request has been cancelled or satisfied when multi-nodes enabled

[jira] [Commented] (YARN-9424) Change getDeclaredMethods to getMethods in FederationClientInterceptor#invokeConcurrent()

35 matches

Site Navigation

Mail list logo

Footer information