[jira] [Commented] (YARN-5909) Remove the agent code in slider and upgrade framework jetty version

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692513#comment-15692513
 ] 

Hadoop QA commented on YARN-5909:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 10 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m  
6s{color} | {color:red} root in yarn-native-services failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m  
5s{color} | {color:red} hadoop-yarn-applications in yarn-native-services 
failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 6s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  1m 
26s{color} | {color:red} hadoop-yarn-services-api in yarn-native-services 
failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m  
5s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services failed. 
{color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red}  0m  
9s{color} | {color:red} hadoop-yarn-services-api in yarn-native-services 
failed. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red}  0m  
5s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services failed. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m  
6s{color} | {color:red} hadoop-yarn-services-api in yarn-native-services 
failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m  
5s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services failed. 
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m  
6s{color} | {color:red} hadoop-yarn-services-api in yarn-native-services 
failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m  
5s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services failed. 
{color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
46s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
29s{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m  
6s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 40s{color} 
| {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications 
generated 36 new + 0 unchanged - 0 fixed = 36 total (was 0) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 24s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications: The patch generated 
330 new + 0 unchanged - 0 fixed = 330 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m  
6s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red}  0m  
7s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m  
7s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
27s{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m  
8s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
13s{color} | {color:green} hadoop-yarn-slider-core in the patch passed. 

[jira] [Created] (YARN-5934) Fix TestTimelineWebServices.testPrimaryFilterNumericString

[Akira Ajisaka (JIRA)]

Akira Ajisaka created YARN-5934:
---

 Summary: Fix TestTimelineWebServices.testPrimaryFilterNumericString
 Key: YARN-5934
 URL: https://issues.apache.org/jira/browse/YARN-5934
 Project: Hadoop YARN
  Issue Type: Bug
  Components: test
Reporter: Akira Ajisaka


{noformat}
Running org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
Tests run: 28, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 43.297 sec <<< 
FAILURE! - in 
org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
testPrimaryFilterNumericString(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
  Time elapsed: 1.209 sec  <<< FAILURE!
java.lang.AssertionError: expected:<0> but was:<3>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testPrimaryFilterNumericString(TestTimelineWebServices.java:348)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4206) Add life time value in Application report and CLI

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692497#comment-15692497
 ] 

Hadoop QA commented on YARN-4206:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
30s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
46s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  4m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 48s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 5 new + 374 unchanged - 2 fixed = 379 total (was 376) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
3s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 34s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m 
20s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
31s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}125m 15s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
|   | hadoop.yarn.server.resourcemanager.TestRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-4206 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840375/YARN-4206.6.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  cc  |
| uname | Linux f33fc337371a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Commented] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692477#comment-15692477
 ] 

Varun Saxena commented on YARN-3884:


Thanks [~bibinchundatt] for the patch.
The log changes made here are not related to your fix. Maybe we can do them in 
a different JIRA as well but they are trivial fixes so I am fine having them 
here as well.

Few nits
# For the log added below in FiCaSchedulerApp, we should have space before 
container on line 683. Also container information should be at the end or 
before node because lines after it correspond to reservation for application. 
Maybe something like Application xxx unreserved container xyz on node
{code}
682 LOG.info("Application " + getApplicationId() + " unreserved "
683 + " on node " + node + "container "
684 + reservedContainer.getContainerId() + ", currently has "
685 + reservedContainers.size() + " at priority "
686 + schedulerKey.getPriority() + "; currentReservation "
687 + this.attemptResourceUsage.getReserved() + " on 
node-label="
688 + node.getPartition());
{code}
# For the other log too in FiCaSchedulerApp, put a comma before 
reservedContainer
# In the test added, in method createReservation we have a comment saying Wait 
for allocation and reservation. We are not waiting for anything here. Maybe say 
something along the lines of send 2 events which leads to reservation.

Other than that +1 from my side.

> RMContainerImpl transition from RESERVED to KILL apphistory status not updated
> --
>
> Key: YARN-3884
> URL: https://issues.apache.org/jira/browse/YARN-3884
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
> Environment: Suse11 Sp3
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>  Labels: oct16-easy
> Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, 
> Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3884.0002.patch, 
> YARN-3884.0003.patch, YARN-3884.0004.patch
>
>
> Setup
> ===
> 1 NM 3072 16 cores each
> Steps to reproduce
> ===
> 1.Submit apps  to Queue 1 with 512 mb 1 core
> 2.Submit apps  to Queue 2 with 512 mb and 5 core
> lots of containers get reserved and unreserved in this case 
> {code}
> 2015-07-02 20:45:31,169 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to 
> RESERVED
> 2015-07-02 20:45:31,170 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: 
> Reserved container  application=application_1435849994778_0002 
> resource= queue=QueueA: capacity=0.4, 
> absoluteCapacity=0.4, usedResources=, 
> usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, 
> numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 
> used= cluster=
> 2015-07-02 20:45:31,170 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, 
> absoluteCapacity=0.4, usedResources=, 
> usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, 
> numContainers=6
> 2015-07-02 20:45:31,170 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.96875 
> absoluteUsedCapacity=0.96875 used= 
> cluster=
> 2015-07-02 20:45:31,191 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to 
> ALLOCATED
> 2015-07-02 20:45:31,191 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf   
> OPERATION=AM Allocated ContainerTARGET=SchedulerApp 
> RESULT=SUCCESS  APPID=application_1435849994778_0001
> CONTAINERID=container_e24_1435849994778_0001_01_14
> 2015-07-02 20:45:31,191 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: 
> Assigned container container_e24_1435849994778_0001_01_14 of capacity 
>  on host host-10-19-92-117:64318, which has 6 
> containers,  used and  available 
> after allocation
> 2015-07-02 20:45:31,191 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: 
> assignedContainer application attempt=appattempt_1435849994778_0001_01 
> container=Container: [ContainerId: 
> 

[jira] [Commented] (YARN-4206) Add life time value in Application report and CLI

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692464#comment-15692464
 ] 

Jian He commented on YARN-4206:
---

+1

> Add life time value in Application report and CLI
> -
>
> Key: YARN-4206
> URL: https://issues.apache.org/jira/browse/YARN-4206
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: nijel
>Assignee: Rohith Sharma K S
> Attachments: YARN-4206.2.patch, YARN-4206.3.patch, YARN-4206.4.patch, 
> YARN-4206.5.patch, YARN-4206.6.patch, YARN-4506.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5649) Add REST endpoints for updating application timeouts

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692445#comment-15692445
 ] 

Jian He commented on YARN-5649:
---

the branch-2 patch looks good to me.

> Add REST endpoints for updating application timeouts
> 
>
> Key: YARN-5649
> URL: https://issues.apache.org/jira/browse/YARN-5649
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: YARN-5649.1.patch, YARN-5649.2.patch, YARN-5649.3.patch, 
> YARN-5649.branch-2.0001.patch, YARN-5649.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5559) Analyse 2.8.0/3.0.0 jdiff reports and fix any issues

[Akira Ajisaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692436#comment-15692436
 ] 

Akira Ajisaka commented on YARN-5559:
-

Ping [~leftnoteasy]. How is this issue going on?

> Analyse 2.8.0/3.0.0 jdiff reports and fix any issues
> 
>
> Key: YARN-5559
> URL: https://issues.apache.org/jira/browse/YARN-5559
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
>  Labels: oct16-easy
> Attachments: YARN-5559.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5920) Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692415#comment-15692415
 ] 

Hudson commented on YARN-5920:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10886 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10886/])
YARN-5920. Fix deadlock in (rohithsharmaks: rev 
e15c20edba1e9a23475ee6a4dfbadbdb8c1f668a)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHA.java


> Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang
> ---
>
> Key: YARN-5920
> URL: https://issues.apache.org/jira/browse/YARN-5920
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Rohith Sharma K S
>Assignee: Varun Saxena
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: ThreadDump.txt, YARN-5920.01.patch, YARN-5920.02.patch
>
>
> In build 
> [linkg|https://builds.apache.org/job/PreCommit-YARN-Build/13986/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt]
>  test case timed out. This need to be investigated.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5572) HBaseTimelineWriterImpl appears to reference a bad property name

[Akira Ajisaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692363#comment-15692363
 ] 

Akira Ajisaka commented on YARN-5572:
-

LGTM, +1.

> HBaseTimelineWriterImpl appears to reference a bad property name
> 
>
> Key: YARN-5572
> URL: https://issues.apache.org/jira/browse/YARN-5572
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Daniel Templeton
>Assignee: Varun Saxena
> Attachments: YARN-5572.01.patch
>
>
> This constructor:
> {code}
>   public HBaseTimelineWriterImpl(Configuration conf) throws IOException {
> super(conf.get("yarn.application.id",
> HBaseTimelineWriterImpl.class.getName()));
>   }
> {code}
> references "yarn.application.id", which is the only reference to this 
> property I could find.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4752) [Umbrella] FairScheduler: Improve preemption

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692343#comment-15692343
 ] 

Hudson commented on YARN-4752:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10885 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10885/])
YARN-4752. Improved preemption in FairScheduler. (kasha) (kasha: rev 
10468529a9b858bd945e7ecb063c9c1438efa474)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueueManagerRealScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSStarvedApps.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerWithMockPreemption.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestSchedulingPolicy.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSContext.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSPreemptionThread.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppStarvation.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FakeSchedulable.java


> [Umbrella] FairScheduler: Improve preemption
> 
>
> Key: YARN-4752
> URL: https://issues.apache.org/jira/browse/YARN-4752
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 2.8.0
>

[jira] [Updated] (YARN-5932) Retrospect moveApplicationToQueue in align with YARN-5611

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-5932:
--
Summary: Retrospect moveApplicationToQueue in align with YARN-5611  (was: 
Retrospect moveToQueue in align with YARN-5611)

> Retrospect moveApplicationToQueue in align with YARN-5611
> -
>
> Key: YARN-5932
> URL: https://issues.apache.org/jira/browse/YARN-5932
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler, resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5932.v0.patch
>
>
> All dynamic api's of an application's state change could follow a general 
> design approach. Currently priority and app timeouts are following this 
> approach all corner cases.
> *Steps*
> - Do a pre-validate check to ensure that changes are fine.
> - Update this information to state-store
> - Perform real move operation and update in-memory data structures.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4206) Add life time value in Application report and CLI

[Rohith Sharma K S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-4206:

Attachment: YARN-4206.6.patch

After having discussion with Jian and other folks, they feel that API should be 
more clarity and easy enough to input values. Providing multiple options makes 
user to bit confusing state. So, going ahed with 2nd suggestion i.e ./yarn 
application -appId $appId -updateLifetime $timeout. 
Attached the patch reflect the same. 

> Add life time value in Application report and CLI
> -
>
> Key: YARN-4206
> URL: https://issues.apache.org/jira/browse/YARN-4206
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: nijel
>Assignee: Rohith Sharma K S
> Attachments: YARN-4206.2.patch, YARN-4206.3.patch, YARN-4206.4.patch, 
> YARN-4206.5.patch, YARN-4206.6.patch, YARN-4506.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5933) ATS stale entries in active directory causes ApplicationNotFoundException in RM

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692171#comment-15692171
 ] 

Sunil G commented on YARN-5933:
---

Yes. Removed the Components tag with v2.

> ATS stale entries in active directory causes ApplicationNotFoundException in 
> RM
> ---
>
> Key: YARN-5933
> URL: https://issues.apache.org/jira/browse/YARN-5933
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>
> On Secure cluster where ATS is down, Tez job submitted will fail while 
> getting TIMELINE_DELEGATION_TOKEN with below exception
> {code}
> 0: jdbc:hive2://kerberos-2.openstacklocal:100> select csmallint from 
> alltypesorc group by csmallint;
> INFO  : Session is already open
> INFO  : Dag name: select csmallint from alltypesor...csmallint(Stage-1)
> INFO  : Tez session was closed. Reopening...
> ERROR : Failed to execute tez graph.
> java.lang.RuntimeException: Failed to connect to timeline server. Connection 
> retries limit exceeded. The posted timeline event may be missing
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:266)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:590)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.getDelegationToken(TimelineClientImpl.java:506)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken(YarnClientImpl.java:349)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.addTimelineDelegationToken(YarnClientImpl.java:330)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:250)
>   at 
> org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72)
>   at org.apache.tez.client.TezClient.start(TezClient.java:409)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeAndOpen(TezSessionPoolManager.java:311)
>   at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:453)
>   at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:180)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1728)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1485)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1126)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1121)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Tez YarnClient has received an applicationID from RM. On Restarting ATS now, 
> ATS tries to get the application report from RM and so RM will throw 
> ApplicationNotFoundException. ATS will keep on requesting and which floods RM.
> {code}
> RM logs:
> 2016-11-23 13:53:57,345 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new 
> applicationId: 5
> 2016-11-23 14:05:04,936 INFO org.apache.hadoop.ipc.Server: IPC Server handler 
> 9 on 8050, call 
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport 
> from 172.26.71.120:37699 Call#26 Retry#0
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application 
> with id 'application_1479897867169_0005' doesn't exist in RM.
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:328)
>   at 
> 

[jira] [Updated] (YARN-5933) ATS stale entries in active directory causes ApplicationNotFoundException in RM

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-5933:
--
Component/s: (was: ATSv2)

> ATS stale entries in active directory causes ApplicationNotFoundException in 
> RM
> ---
>
> Key: YARN-5933
> URL: https://issues.apache.org/jira/browse/YARN-5933
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>
> On Secure cluster where ATS is down, Tez job submitted will fail while 
> getting TIMELINE_DELEGATION_TOKEN with below exception
> {code}
> 0: jdbc:hive2://kerberos-2.openstacklocal:100> select csmallint from 
> alltypesorc group by csmallint;
> INFO  : Session is already open
> INFO  : Dag name: select csmallint from alltypesor...csmallint(Stage-1)
> INFO  : Tez session was closed. Reopening...
> ERROR : Failed to execute tez graph.
> java.lang.RuntimeException: Failed to connect to timeline server. Connection 
> retries limit exceeded. The posted timeline event may be missing
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:266)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:590)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.getDelegationToken(TimelineClientImpl.java:506)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken(YarnClientImpl.java:349)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.addTimelineDelegationToken(YarnClientImpl.java:330)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:250)
>   at 
> org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72)
>   at org.apache.tez.client.TezClient.start(TezClient.java:409)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeAndOpen(TezSessionPoolManager.java:311)
>   at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:453)
>   at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:180)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1728)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1485)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1126)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1121)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Tez YarnClient has received an applicationID from RM. On Restarting ATS now, 
> ATS tries to get the application report from RM and so RM will throw 
> ApplicationNotFoundException. ATS will keep on requesting and which floods RM.
> {code}
> RM logs:
> 2016-11-23 13:53:57,345 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new 
> applicationId: 5
> 2016-11-23 14:05:04,936 INFO org.apache.hadoop.ipc.Server: IPC Server handler 
> 9 on 8050, call 
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport 
> from 172.26.71.120:37699 Call#26 Retry#0
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application 
> with id 'application_1479897867169_0005' doesn't exist in RM.
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:328)
>   at 
> 

[jira] [Updated] (YARN-5649) Add REST endpoints for updating application timeouts

[Rohith Sharma K S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-5649:

Target Version/s: 2.9.0  (was: 2.8.0)

> Add REST endpoints for updating application timeouts
> 
>
> Key: YARN-5649
> URL: https://issues.apache.org/jira/browse/YARN-5649
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: YARN-5649.1.patch, YARN-5649.2.patch, YARN-5649.3.patch, 
> YARN-5649.branch-2.0001.patch, YARN-5649.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5649) Add REST endpoints for updating application timeouts

[Rohith Sharma K S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-5649:

Fix Version/s: (was: 2.8.0)

> Add REST endpoints for updating application timeouts
> 
>
> Key: YARN-5649
> URL: https://issues.apache.org/jira/browse/YARN-5649
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: YARN-5649.1.patch, YARN-5649.2.patch, YARN-5649.3.patch, 
> YARN-5649.branch-2.0001.patch, YARN-5649.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5933) ATS stale entries in active directory causes ApplicationNotFoundException in RM

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692165#comment-15692165
 ] 

Naganarasimha G R commented on YARN-5933:
-

isnt this issue of ATS 1.5 ?

> ATS stale entries in active directory causes ApplicationNotFoundException in 
> RM
> ---
>
> Key: YARN-5933
> URL: https://issues.apache.org/jira/browse/YARN-5933
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: ATSv2
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>
> On Secure cluster where ATS is down, Tez job submitted will fail while 
> getting TIMELINE_DELEGATION_TOKEN with below exception
> {code}
> 0: jdbc:hive2://kerberos-2.openstacklocal:100> select csmallint from 
> alltypesorc group by csmallint;
> INFO  : Session is already open
> INFO  : Dag name: select csmallint from alltypesor...csmallint(Stage-1)
> INFO  : Tez session was closed. Reopening...
> ERROR : Failed to execute tez graph.
> java.lang.RuntimeException: Failed to connect to timeline server. Connection 
> retries limit exceeded. The posted timeline event may be missing
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:266)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:590)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.getDelegationToken(TimelineClientImpl.java:506)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken(YarnClientImpl.java:349)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.addTimelineDelegationToken(YarnClientImpl.java:330)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:250)
>   at 
> org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72)
>   at org.apache.tez.client.TezClient.start(TezClient.java:409)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeAndOpen(TezSessionPoolManager.java:311)
>   at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:453)
>   at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:180)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1728)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1485)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1126)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1121)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Tez YarnClient has received an applicationID from RM. On Restarting ATS now, 
> ATS tries to get the application report from RM and so RM will throw 
> ApplicationNotFoundException. ATS will keep on requesting and which floods RM.
> {code}
> RM logs:
> 2016-11-23 13:53:57,345 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new 
> applicationId: 5
> 2016-11-23 14:05:04,936 INFO org.apache.hadoop.ipc.Server: IPC Server handler 
> 9 on 8050, call 
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport 
> from 172.26.71.120:37699 Call#26 Retry#0
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application 
> with id 'application_1479897867169_0005' doesn't exist in RM.
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:328)
>

[jira] [Updated] (YARN-5649) Add REST endpoints for updating application timeouts

[Rohith Sharma K S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-5649:

Attachment: YARN-5649.branch-2.0001.patch

Updated branch-2 patch. cc:/[~jianhe] 

> Add REST endpoints for updating application timeouts
> 
>
> Key: YARN-5649
> URL: https://issues.apache.org/jira/browse/YARN-5649
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 2.8.0
>
> Attachments: YARN-5649.1.patch, YARN-5649.2.patch, YARN-5649.3.patch, 
> YARN-5649.branch-2.0001.patch, YARN-5649.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-5933) ATS stale entries in active directory causes ApplicationNotFoundException in RM

[Prabhu Joseph (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prabhu Joseph reassigned YARN-5933:
---

Assignee: Prabhu Joseph

> ATS stale entries in active directory causes ApplicationNotFoundException in 
> RM
> ---
>
> Key: YARN-5933
> URL: https://issues.apache.org/jira/browse/YARN-5933
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: ATSv2
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>
> On Secure cluster where ATS is down, Tez job submitted will fail while 
> getting TIMELINE_DELEGATION_TOKEN with below exception
> {code}
> 0: jdbc:hive2://kerberos-2.openstacklocal:100> select csmallint from 
> alltypesorc group by csmallint;
> INFO  : Session is already open
> INFO  : Dag name: select csmallint from alltypesor...csmallint(Stage-1)
> INFO  : Tez session was closed. Reopening...
> ERROR : Failed to execute tez graph.
> java.lang.RuntimeException: Failed to connect to timeline server. Connection 
> retries limit exceeded. The posted timeline event may be missing
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:266)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:590)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.getDelegationToken(TimelineClientImpl.java:506)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken(YarnClientImpl.java:349)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.addTimelineDelegationToken(YarnClientImpl.java:330)
>   at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:250)
>   at 
> org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72)
>   at org.apache.tez.client.TezClient.start(TezClient.java:409)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeAndOpen(TezSessionPoolManager.java:311)
>   at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:453)
>   at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:180)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1728)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1485)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1126)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1121)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Tez YarnClient has received an applicationID from RM. On Restarting ATS now, 
> ATS tries to get the application report from RM and so RM will throw 
> ApplicationNotFoundException. ATS will keep on requesting and which floods RM.
> {code}
> RM logs:
> 2016-11-23 13:53:57,345 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new 
> applicationId: 5
> 2016-11-23 14:05:04,936 INFO org.apache.hadoop.ipc.Server: IPC Server handler 
> 9 on 8050, call 
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport 
> from 172.26.71.120:37699 Call#26 Retry#0
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application 
> with id 'application_1479897867169_0005' doesn't exist in RM.
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:328)
>   at 
> 

[jira] [Created] (YARN-5933) ATS stale entries in active directory causes ApplicationNotFoundException in RM

[Prabhu Joseph (JIRA)]

Prabhu Joseph created YARN-5933:
---

 Summary: ATS stale entries in active directory causes 
ApplicationNotFoundException in RM
 Key: YARN-5933
 URL: https://issues.apache.org/jira/browse/YARN-5933
 Project: Hadoop YARN
  Issue Type: Bug
  Components: ATSv2
Affects Versions: 2.7.3
Reporter: Prabhu Joseph


On Secure cluster where ATS is down, Tez job submitted will fail while getting 
TIMELINE_DELEGATION_TOKEN with below exception

{code}
0: jdbc:hive2://kerberos-2.openstacklocal:100> select csmallint from 
alltypesorc group by csmallint;
INFO  : Session is already open
INFO  : Dag name: select csmallint from alltypesor...csmallint(Stage-1)
INFO  : Tez session was closed. Reopening...
ERROR : Failed to execute tez graph.
java.lang.RuntimeException: Failed to connect to timeline server. Connection 
retries limit exceeded. The posted timeline event may be missing
at 
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:266)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:590)
at 
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.getDelegationToken(TimelineClientImpl.java:506)
at 
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken(YarnClientImpl.java:349)
at 
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.addTimelineDelegationToken(YarnClientImpl.java:330)
at 
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:250)
at 
org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72)
at org.apache.tez.client.TezClient.start(TezClient.java:409)
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeAndOpen(TezSessionPoolManager.java:311)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:453)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:180)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1728)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1485)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1126)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1121)
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
at 
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at 
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at 
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}

Tez YarnClient has received an applicationID from RM. On Restarting ATS now, 
ATS tries to get the application report from RM and so RM will throw 
ApplicationNotFoundException. ATS will keep on requesting and which floods RM.

{code}
RM logs:
2016-11-23 13:53:57,345 INFO 
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new 
applicationId: 5
2016-11-23 14:05:04,936 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 
on 8050, call 
org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport 
from 172.26.71.120:37699 Call#26 Retry#0
org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application 
with id 'application_1479897867169_0005' doesn't exist in RM.
at 
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:328)
at 
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:175)
at 
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:417)
at 

[jira] [Reopened] (YARN-5649) Add REST endpoints for updating application timeouts

[Rohith Sharma K S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S reopened YARN-5649:
-

Reopening the issue for handling revert, and uploading a new patch to branch-2. 

> Add REST endpoints for updating application timeouts
> 
>
> Key: YARN-5649
> URL: https://issues.apache.org/jira/browse/YARN-5649
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 2.8.0
>
> Attachments: YARN-5649.1.patch, YARN-5649.2.patch, YARN-5649.3.patch, 
> YARN-5649.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5649) Add REST endpoints for updating application timeouts

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692089#comment-15692089
 ] 

Rohith Sharma K S commented on YARN-5649:
-

yes, let us revert the changes as many conflicts are there to resolve. I will 
upload a new patch for branch-2 again. 

> Add REST endpoints for updating application timeouts
> 
>
> Key: YARN-5649
> URL: https://issues.apache.org/jira/browse/YARN-5649
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 2.8.0
>
> Attachments: YARN-5649.1.patch, YARN-5649.2.patch, YARN-5649.3.patch, 
> YARN-5649.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5774) MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set yarn.scheduler.minimum-allocation-mb to 0.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692073#comment-15692073
 ] 

Hadoop QA commented on YARN-5774:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
23s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 47s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 1 new + 344 unchanged - 12 fixed = 345 total (was 356) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
31s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 0 new + 922 unchanged - 13 fixed = 922 total (was 935) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
30s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
23s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m 
18s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 42s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5774 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840362/YARN-5774.007.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 8e764120d7ad 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Commented] (YARN-5694) ZKRMStateStore can prevent the transition to standby in branch-2.7 if the ZK node is unreachable

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692065#comment-15692065
 ] 

Daniel Templeton commented on YARN-5694:


I ran {{TestClientRMTokens}} 500 times with just branch-2.7 and 500 times with 
branch-2.7 + this patch.  In neither case did I encounter any test failures.

> ZKRMStateStore can prevent the transition to standby in branch-2.7 if the ZK 
> node is unreachable
> 
>
> Key: YARN-5694
> URL: https://issues.apache.org/jira/browse/YARN-5694
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.3
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>  Labels: oct16-medium
> Attachments: YARN-5694.001.patch, YARN-5694.002.patch, 
> YARN-5694.003.patch, YARN-5694.004.patch, YARN-5694.004.patch, 
> YARN-5694.005.patch, YARN-5694.006.patch, YARN-5694.007.patch, 
> YARN-5694.008.patch, YARN-5694.branch-2.7.001.patch, 
> YARN-5694.branch-2.7.002.patch, YARN-5694.branch-2.7.004.patch, 
> YARN-5694.branch-2.7.005.patch
>
>
> {{ZKRMStateStore.doStoreMultiWithRetries()}} holds the lock while trying to 
> talk to ZK.  If the connection fails, it will retry while still holding the 
> lock.  The retries are intended to be strictly time limited, but in the case 
> that the ZK node is unreachable, the time limit fails, resulting in the 
> thread holding the lock for over an hour.  Transitioning the RM to standby 
> requires that same lock, so in exactly the case that the RM should be 
> transitioning to standby, the {{VerifyActiveStatusThread}} blocks it from 
> happening.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5292) Support for PAUSED container state

[Hitesh Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692011#comment-15692011
 ] 

Hitesh Sharma commented on YARN-5292:
-

Hi [~jianhe], apologies for the late response. It seems that [YARN-4876] adds 
the functionality to do what you are describing. Please let me know if you have 
something else in mind here. Thanks!

> Support for PAUSED container state
> --
>
> Key: YARN-5292
> URL: https://issues.apache.org/jira/browse/YARN-5292
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Hitesh Sharma
>Assignee: Hitesh Sharma
> Attachments: YARN-5292.001.patch, YARN-5292.002.patch, 
> YARN-5292.003.patch, yarn-5292.pdf
>
>
> YARN-2877 introduced OPPORTUNISTIC containers, and YARN-5216 proposes to add 
> capability to customize how OPPORTUNISTIC containers get preempted.
> In this JIRA we propose introducing a PAUSED container state.
> When a running container gets preempted, it enters the PAUSED state, where it 
> remains until resources get freed up on the node then the preempted container 
> can resume to the running state.
>  
> One scenario where this capability is useful is work preservation. How 
> preemption is done, and whether the container supports it, is implementation 
> specific.
> For instance, if the container is a virtual machine, then preempt would pause 
> the VM and resume would restore it back to the running state.
> If the container doesn't support preemption, then preempt would default to 
> killing the container. 
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5774) MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set yarn.scheduler.minimum-allocation-mb to 0.

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691929#comment-15691929
 ] 

Yufei Gu commented on YARN-5774:


[~templedf], I've uploaded the patch 007 for all your comments except the last 
one since the first case in {{testNormalize}} covers that. 
For YARN-5795, this patch will solve the issue incidentally but we can do more 
in that JIRA as I commented in it. 

> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set 
> yarn.scheduler.minimum-allocation-mb to 0.
> 
>
> Key: YARN-5774
> URL: https://issues.apache.org/jira/browse/YARN-5774
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>  Labels: oct16-easy
> Attachments: YARN-5774.001.patch, YARN-5774.002.patch, 
> YARN-5774.003.patch, YARN-5774.004.patch, YARN-5774.005.patch, 
> YARN-5774.006.patch, YARN-5774.007.patch
>
>
> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler 
> because there is no resource request for the AM. This happened when you 
> configure {{yarn.scheduler.minimum-allocation-mb}} to zero.
> The problem is in the code used by both Capacity Scheduler and Fair 
> Scheduler. {{scheduler.increment-allocation-mb}} is a concept in FS, but not 
> CS. So the common code in class RMAppManager passes the 
> {{yarn.scheduler.minimum-allocation-mb}} as incremental one because there is 
> no incremental one for CS when it tried to normalize the resource requests.
> {code}
>  SchedulerUtils.normalizeRequest(amReq, scheduler.getResourceCalculator(),
>   scheduler.getClusterResource(),
>   scheduler.getMinimumResourceCapability(),
>   scheduler.getMaximumResourceCapability(),
>   scheduler.getMinimumResourceCapability());  --> incrementResource 
> should be passed here.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5774) MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set yarn.scheduler.minimum-allocation-mb to 0.

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-5774:
---
Attachment: YARN-5774.007.patch

> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set 
> yarn.scheduler.minimum-allocation-mb to 0.
> 
>
> Key: YARN-5774
> URL: https://issues.apache.org/jira/browse/YARN-5774
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>  Labels: oct16-easy
> Attachments: YARN-5774.001.patch, YARN-5774.002.patch, 
> YARN-5774.003.patch, YARN-5774.004.patch, YARN-5774.005.patch, 
> YARN-5774.006.patch, YARN-5774.007.patch
>
>
> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler 
> because there is no resource request for the AM. This happened when you 
> configure {{yarn.scheduler.minimum-allocation-mb}} to zero.
> The problem is in the code used by both Capacity Scheduler and Fair 
> Scheduler. {{scheduler.increment-allocation-mb}} is a concept in FS, but not 
> CS. So the common code in class RMAppManager passes the 
> {{yarn.scheduler.minimum-allocation-mb}} as incremental one because there is 
> no incremental one for CS when it tried to normalize the resource requests.
> {code}
>  SchedulerUtils.normalizeRequest(amReq, scheduler.getResourceCalculator(),
>   scheduler.getClusterResource(),
>   scheduler.getMinimumResourceCapability(),
>   scheduler.getMaximumResourceCapability(),
>   scheduler.getMinimumResourceCapability());  --> incrementResource 
> should be passed here.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5649) Add REST endpoints for updating application timeouts

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691854#comment-15691854
 ] 

Mingliang Liu commented on YARN-5649:
-

{{branch-2}} is broken because of this?

> Add REST endpoints for updating application timeouts
> 
>
> Key: YARN-5649
> URL: https://issues.apache.org/jira/browse/YARN-5649
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 2.8.0
>
> Attachments: YARN-5649.1.patch, YARN-5649.2.patch, YARN-5649.3.patch, 
> YARN-5649.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5649) Add REST endpoints for updating application timeouts

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691796#comment-15691796
 ] 

Hudson commented on YARN-5649:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10882 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10882/])
YARN-5649. Add REST endpoints for updating application timeouts. (jianhe: rev 
1f12867a69544a1642aa986d4f9a8249be495434)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAuditLogger.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesAppsModification.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppTimeoutsInfo.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppTimeoutInfo.java


> Add REST endpoints for updating application timeouts
> 
>
> Key: YARN-5649
> URL: https://issues.apache.org/jira/browse/YARN-5649
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 2.8.0
>
> Attachments: YARN-5649.1.patch, YARN-5649.2.patch, YARN-5649.3.patch, 
> YARN-5649.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5929) Missing scheduling policy in the FS queue metric.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691782#comment-15691782
 ] 

Hadoop QA commented on YARN-5929:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 22s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m 
43s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 45s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5929 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840331/YARN-5929.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux ed7fa97f7150 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0de0c32 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/14068/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14068/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14068/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Missing scheduling policy in the FS queue metric. 
> --
>
> Key: YARN-5929
>

[jira] [Commented] (YARN-5795) FairScheduler set AppMaster vcores didn't work

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691743#comment-15691743
 ] 

Yufei Gu commented on YARN-5795:


Yes. I found this issue as well. No matter how many vcores you give it to 
resource request, {{DefaultResourceCalculator}} will assign 1 to it while 
normalizing. 
It is hard to say if this is bug for {{DefaultResourceCalculator}} since it 
depends on how/when we use it.
This is more about an issue of fair scheduler. In FairScheduler, we use the 
{{DominantResourceCalculator}} in the most case, but the weird thing is if your 
invoke {{fairScheduler.getResourceCalculator()}}, you will get a 
{{DefaultResourceCalculator}}. That doesn't sound reasonable to me. We should 
investigate and make some changes.
YARN-5774 can solve this issue incidentally. Maybe we could focus on the what 
should {{fairScheduler.getResourceCalculator()}} return in this JIRA.

> FairScheduler set AppMaster vcores didn't work
> --
>
> Key: YARN-5795
> URL: https://issues.apache.org/jira/browse/YARN-5795
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: tangshangwen
> Attachments: 0001-YARN-5795.patch
>
>
> In our cluster, we use the linux container, I would like to increase the 
> number of cpu to get more CPU time slice, but it did not take effect， i set 
> yarn.app.mapreduce.am.resource.cpu-vcores = 2 ,but i found the 
> resourcemanager log
> {noformat}
> [2016-10-27T16:36:37.280 08:00] [INFO] 
> resourcemanager.scheduler.SchedulerNode.allocateContainer(SchedulerNode.java 
> 153) [ResourceManager Event Processor] : Assigned container 
> container_1477059529836_336635_01_01 of capacity 
> {noformat}
> Because scheduler.getResourceCalculator() only computes memory
> {code:title=RMAppManager.java|borderStyle=solid}
>  private ResourceRequest validateAndCreateResourceRequest(
>   ApplicationSubmissionContext submissionContext, boolean isRecovery)
>   throws InvalidResourceRequestException {
> ...
>   SchedulerUtils.normalizeRequest(amReq, 
> scheduler.getResourceCalculator(),
>   scheduler.getClusterResource(),
>   scheduler.getMinimumResourceCapability(),
>   scheduler.getMaximumResourceCapability(),
>   scheduler.getMinimumResourceCapability());
> ...
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5901) Fix race condition in TestGetGroups beforeclass setup()

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691742#comment-15691742
 ] 

Daniel Templeton commented on YARN-5901:


Two tiny nits:

# In your service listener, you should probably call 
{{service.getServiceState()}} instead, and
# In the test, why is the {{rmStarted}} variable final?

> Fix race condition in TestGetGroups beforeclass setup()
> ---
>
> Key: YARN-5901
> URL: https://issues.apache.org/jira/browse/YARN-5901
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>  Labels: unittest
> Attachments: YARN-5901.02.patch, yarn5901.001.patch
>
>
> In TestGetGroups, the class-level setup method spins up, in a child thread, a 
> resource manager that Yarn clients can talk to. But it checks whether the 
> resource manager is fully started by doing resourcemanager.getServiceState() 
> == STATE.STARTED. This is not reliable since resourcemanager.start() will 
> first trigger service state change in RM, and then starts up all the services 
> added to RM. We need to wait for RM to fully start before YARN clients  can 
> send requests. Otherwise, the tests can fail due to "connection refused"  
> exception when the main thread sends out client requests to RM and if the RPC 
> server has not fired up in the child thread.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5901) Fix race condition in TestGetGroups beforeclass setup()

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691709#comment-15691709
 ] 

Hadoop QA commented on YARN-5901:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
53s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: 
The patch generated 0 new + 1 unchanged - 1 fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 
15s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 21s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5901 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840333/YARN-5901.02.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux ff9445c028f3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0de0c32 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14066/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14066/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Fix race condition in TestGetGroups beforeclass setup()
> ---
>
> Key: YARN-5901
> URL: https://issues.apache.org/jira/browse/YARN-5901
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>  Labels: unittest
> Attachments: YARN-5901.02.patch, 

[jira] [Commented] (YARN-5903) Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl beforeclass setup method

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691708#comment-15691708
 ] 

Hadoop QA commented on YARN-5903:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 
12s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 11s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5903 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840332/YARN-5903.02.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 2f75562cecab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0de0c32 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14067/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14067/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl 
> beforeclass setup method
> 
>
> Key: YARN-5903
> URL: https://issues.apache.org/jira/browse/YARN-5903
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-5903.02.patch, yarn5903.001.patch
>
>
> This is essentially the 

[jira] [Commented] (YARN-5909) Remove the agent code in slider and upgrade framework jetty version

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691692#comment-15691692
 ] 

Jian He commented on YARN-5909:
---

This patch removed all the agent related code in slider AM, also changed the 
jetty version to 9
AgentKeys is not yet removed. It's being referenced in a lot of other places.
Plan to rename it after the current patch looks fine.

[~billie.rinaldi] [~gsaha], could you take a look ?

> Remove the agent code in slider and upgrade framework jetty version 
> 
>
> Key: YARN-5909
> URL: https://issues.apache.org/jira/browse/YARN-5909
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
> Attachments: YARN-5909-yarn-native-services.01.patch
>
>
> Hadoop core has upgraded jetty version to 9 ,
> The framework (slider AM) also needs to upgrade the jetty version
> Problem is that some legacy agent code uses classes which only exist in old 
> jetty. Probably it's the time to remove all the agent related code ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5909) Remove the agent code in slider and upgrade framework jetty version

[Jian He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5909:
--
Attachment: YARN-5909-yarn-native-services.01.patch

> Remove the agent code in slider and upgrade framework jetty version 
> 
>
> Key: YARN-5909
> URL: https://issues.apache.org/jira/browse/YARN-5909
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
> Attachments: YARN-5909-yarn-native-services.01.patch
>
>
> Hadoop core has upgraded jetty version to 9 ,
> The framework (slider AM) also needs to upgrade the jetty version
> Problem is that some legacy agent code uses classes which only exist in old 
> jetty. Probably it's the time to remove all the agent related code ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5883) Avoid or eliminate expensive YARN get all applications call

[Billie Rinaldi (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691660#comment-15691660
 ] 

Billie Rinaldi commented on YARN-5883:
--

+1, the new patch looks good.

> Avoid or eliminate expensive YARN get all applications call
> ---
>
> Key: YARN-5883
> URL: https://issues.apache.org/jira/browse/YARN-5883
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Gour Saha
> Fix For: yarn-native-services
>
> Attachments: YARN-5883-yarn-native-services.001.patch, 
> YARN-5883-yarn-native-services.002.patch
>
>
> YARN _getAllApplications()_ call with no param or with application types 
> _getApplications(Set applicationTypes)_ are very costly in a cluster 
> with hundreds or thousands of Slider apps running.
> YARN now has the API to filter by application tags - YARN-4491. We should be 
> able to use this to avoid these expensive calls.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5909) Remove the agent code in slider and upgrade framework jetty version

[Jian He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5909:
--
Summary: Remove the agent code in slider and upgrade framework jetty 
version   (was: Upgrade framework jetty version )

> Remove the agent code in slider and upgrade framework jetty version 
> 
>
> Key: YARN-5909
> URL: https://issues.apache.org/jira/browse/YARN-5909
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>
> Hadoop core has upgraded jetty version to 9 ,
> The framework (slider AM) also needs to upgrade the jetty version
> Problem is that some legacy agent code uses classes which only exist in old 
> jetty. Probably it's the time to remove all the agent related code ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5901) Fix race condition in TestGetGroups beforeclass setup()

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated YARN-5901:
-
Attachment: YARN-5901.02.patch

> Fix race condition in TestGetGroups beforeclass setup()
> ---
>
> Key: YARN-5901
> URL: https://issues.apache.org/jira/browse/YARN-5901
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>  Labels: unittest
> Attachments: YARN-5901.02.patch, yarn5901.001.patch
>
>
> In TestGetGroups, the class-level setup method spins up, in a child thread, a 
> resource manager that Yarn clients can talk to. But it checks whether the 
> resource manager is fully started by doing resourcemanager.getServiceState() 
> == STATE.STARTED. This is not reliable since resourcemanager.start() will 
> first trigger service state change in RM, and then starts up all the services 
> added to RM. We need to wait for RM to fully start before YARN clients  can 
> send requests. Otherwise, the tests can fail due to "connection refused"  
> exception when the main thread sends out client requests to RM and if the RPC 
> server has not fired up in the child thread.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5901) Fix race condition in TestGetGroups beforeclass setup()

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691579#comment-15691579
 ] 

Haibo Chen commented on YARN-5901:
--

Thanks [~templedf] for your review! Will update the patch shortly.

> Fix race condition in TestGetGroups beforeclass setup()
> ---
>
> Key: YARN-5901
> URL: https://issues.apache.org/jira/browse/YARN-5901
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>  Labels: unittest
> Attachments: yarn5901.001.patch
>
>
> In TestGetGroups, the class-level setup method spins up, in a child thread, a 
> resource manager that Yarn clients can talk to. But it checks whether the 
> resource manager is fully started by doing resourcemanager.getServiceState() 
> == STATE.STARTED. This is not reliable since resourcemanager.start() will 
> first trigger service state change in RM, and then starts up all the services 
> added to RM. We need to wait for RM to fully start before YARN clients  can 
> send requests. Otherwise, the tests can fail due to "connection refused"  
> exception when the main thread sends out client requests to RM and if the RPC 
> server has not fired up in the child thread.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5903) Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl beforeclass setup method

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated YARN-5903:
-
Attachment: YARN-5903.02.patch

> Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl 
> beforeclass setup method
> 
>
> Key: YARN-5903
> URL: https://issues.apache.org/jira/browse/YARN-5903
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-5903.02.patch, yarn5903.001.patch
>
>
> This is essentially the same race condition as in YARN-5901, that is, 
> resourcemanager.getServiceState() == STATE.STARTED does not guarantee 
> resource manager is fully started.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5929) Missing scheduling policy in the FS queue metric.

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-5929:
---
Attachment: YARN-5929.001.patch

> Missing scheduling policy in the FS queue metric. 
> --
>
> Key: YARN-5929
> URL: https://issues.apache.org/jira/browse/YARN-5929
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5929.001.patch
>
>
> It should be there since YARN-4878. But it doesn't. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5929) Missing scheduling policy in the queue metric.

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-5929:
---
Summary: Missing scheduling policy in the queue metric.   (was: Miss 
scheduling policy in the queue metric. )

> Missing scheduling policy in the queue metric. 
> ---
>
> Key: YARN-5929
> URL: https://issues.apache.org/jira/browse/YARN-5929
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> It should be there since YARN-4878. But it doesn't. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5929) Missing scheduling policy in the FS queue metric.

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-5929:
---
Summary: Missing scheduling policy in the FS queue metric.   (was: Missing 
scheduling policy in the queue metric. )

> Missing scheduling policy in the FS queue metric. 
> --
>
> Key: YARN-5929
> URL: https://issues.apache.org/jira/browse/YARN-5929
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> It should be there since YARN-4878. But it doesn't. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4206) Add life time value in Application report and CLI

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691535#comment-15691535
 ] 

Jian He commented on YARN-4206:
---

would a single option -updateLifetime be simper ? having one more option makes 
user memorize one more option... 

> Add life time value in Application report and CLI
> -
>
> Key: YARN-4206
> URL: https://issues.apache.org/jira/browse/YARN-4206
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: nijel
>Assignee: Rohith Sharma K S
> Attachments: YARN-4206.2.patch, YARN-4206.3.patch, YARN-4206.4.patch, 
> YARN-4206.5.patch, YARN-4506.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5903) Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl beforeclass setup method

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691528#comment-15691528
 ] 

Haibo Chen commented on YARN-5903:
--

My bad. I incorrectly interpret the CountDownLatch API. WIll update the patch 
incorporate  your comments.

> Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl 
> beforeclass setup method
> 
>
> Key: YARN-5903
> URL: https://issues.apache.org/jira/browse/YARN-5903
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: yarn5903.001.patch
>
>
> This is essentially the same race condition as in YARN-5901, that is, 
> resourcemanager.getServiceState() == STATE.STARTED does not guarantee 
> resource manager is fully started.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5649) Add REST endpoints for updating application timeouts

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691478#comment-15691478
 ] 

Jian He commented on YARN-5649:
---

looks good, committing

> Add REST endpoints for updating application timeouts
> 
>
> Key: YARN-5649
> URL: https://issues.apache.org/jira/browse/YARN-5649
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: YARN-5649.1.patch, YARN-5649.2.patch, YARN-5649.3.patch, 
> YARN-5649.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5877) Allow all nm-whitelist-env to get overridden during launch

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691396#comment-15691396
 ] 

Varun Saxena commented on YARN-5877:


Thanks [~bibinchundatt] for the patch. The approach as such seems fine and is 
consistent with previous implementation. Do we however need to provide a 
facility for container launch context env to take precedence for some 
containers and take default from docker env for some others ? Anyways, that 
would need more discussion regarding if there is any use case for this.

Coming to the patch.
# We need to add this config in yarn-default.xml
# getWhitelistEnv can be protected
# Name getWhitelistEnv -> getEnvToBeWhitelisted ? Also javadoc should clearly 
indicate what this method is meant for and why derived classes should override 
it.
# There is an unused import in ContainerExecutor.
# In LinuxContainerExecutor#getWhitelistEnv, while iterating over the envs 
configured, we should trim them before adding them to the set.

I would like to hear from guys who worked in the area of docker support in YARN 
as well.

> Allow all nm-whitelist-env to get overridden during launch
> --
>
> Key: YARN-5877
> URL: https://issues.apache.org/jira/browse/YARN-5877
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: Dockerfile, YARN-5877.0001.patch, YARN-5877.0002.patch, 
> YARN-5877.0003.patch, bootstrap.sh, yarn-site.xml
>
>
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values 
> should  containers may override rather than use NodeManager's default.
> {code}
>   
> Environment variables that containers may override rather 
> than use NodeManager's default.
> yarn.nodemanager.env-whitelist
> 
> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME
>   
> {code}
> But only the following containers can override
> {code}
> whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
> whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5739) Provide timeline reader API to list available timeline entity types for one application

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691293#comment-15691293
 ] 

Varun Saxena commented on YARN-5739:


Ok...Will wait.

> Provide timeline reader API to list available timeline entity types for one 
> application
> ---
>
> Key: YARN-5739
> URL: https://issues.apache.org/jira/browse/YARN-5739
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Li Lu
>Assignee: Li Lu
> Attachments: YARN-5739-YARN-5355.001.patch, 
> YARN-5739-YARN-5355.002.patch, YARN-5739-YARN-5355.003.patch, 
> YARN-5739-YARN-5355.004.patch
>
>
> Right now we only show a part of available timeline entity data in the new 
> YARN UI. However, some data (especially library specific data) are not 
> possible to be queried out by the web UI. It will be appealing for the UI to 
> provide an "entity browser" for each YARN application. Actually, simply 
> dumping out available timeline entities (with proper pagination, of course) 
> would be pretty helpful for UI users. 
> On timeline side, we're not far away from this goal. Right now I believe the 
> only thing missing is to list all available entity types within one 
> application. The challenge here is that we're not storing this data for each 
> application, but given this kind of call is relatively rare (compare to 
> writes and updates) we can perform some scanning during the read time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5649) Add REST endpoints for updating application timeouts

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691281#comment-15691281
 ] 

Hadoop QA commented on YARN-5649:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 22s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 3 new + 227 unchanged - 1 fixed = 230 total (was 228) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 42m 21s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 27s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
|   | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5649 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840298/YARN-5649.3.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 2352cf43d527 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 005850b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/14065/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/14065/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14065/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| 

[jira] [Commented] (YARN-5739) Provide timeline reader API to list available timeline entity types for one application

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691232#comment-15691232
 ] 

Sangjin Lee commented on YARN-5739:
---

Can we wait until next week? Most of us are for the holiday this week...

On Wed, Nov 23, 2016 at 11:48 AM, Varun Saxena (JIRA) 



> Provide timeline reader API to list available timeline entity types for one 
> application
> ---
>
> Key: YARN-5739
> URL: https://issues.apache.org/jira/browse/YARN-5739
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Li Lu
>Assignee: Li Lu
> Attachments: YARN-5739-YARN-5355.001.patch, 
> YARN-5739-YARN-5355.002.patch, YARN-5739-YARN-5355.003.patch, 
> YARN-5739-YARN-5355.004.patch
>
>
> Right now we only show a part of available timeline entity data in the new 
> YARN UI. However, some data (especially library specific data) are not 
> possible to be queried out by the web UI. It will be appealing for the UI to 
> provide an "entity browser" for each YARN application. Actually, simply 
> dumping out available timeline entities (with proper pagination, of course) 
> would be pretty helpful for UI users. 
> On timeline side, we're not far away from this goal. Right now I believe the 
> only thing missing is to list all available entity types within one 
> application. The challenge here is that we're not storing this data for each 
> application, but given this kind of call is relatively rare (compare to 
> writes and updates) we can perform some scanning during the read time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5849) Automatically create YARN control group for pre-mounted cgroups

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691227#comment-15691227
 ] 

Daniel Templeton commented on YARN-5849:


While we wait for [~bibinchundatt] to review the latest patch, I have a few 
minor comments:

* Thanks for fixing the javadocs!
* In {{CGroupsHandler}}, let's leave the {{MTAB_FILE}} constant there for use 
in the new constructor.
* The new constructor needs javadocs.
* Looks to me like {{getErrorWithDetails()}} should just return the {{String}}.
* In {{getErrorWithDetails()}} you may want to use 
{{UserGroupInformation.getCurrentUser()}} instead of the system property.
* Missing a {{@param}} in the javadoc for 
{{testPreMountedControllerInitialization()}}
* It would be good to have messages in all your asserts.
* In the docs, it would be good to be clearer about the new behavior, e.g. that 
the mount point will be created if it doesn't exist, that it needs write 
permissions, etc.  Basically a brain dump to help the poor admin who's trying 
to debug a broken cluster.
* Does {{yarn-defaults.xml}} also need to be updated?

> Automatically create YARN control group for pre-mounted cgroups
> ---
>
> Key: YARN-5849
> URL: https://issues.apache.org/jira/browse/YARN-5849
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
> Attachments: YARN-5849.000.patch, YARN-5849.001.patch, 
> YARN-5849.002.patch, YARN-5849.003.patch
>
>
> Yarn can be launched with linux-container-executor.cgroups.mount set to 
> false. It will search for the cgroup mount paths set up by the administrator 
> parsing the /etc/mtab file. You can also specify 
> resource.percentage-physical-cpu-limit to limit the CPU resources assigned to 
> containers.
> linux-container-executor.cgroups.hierarchy is the root of the settings of all 
> YARN containers. If this is specified but not created YARN will fail at 
> startup:
> Caused by: java.io.FileNotFoundException: 
> /cgroups/cpu/hadoop-yarn/cpu.cfs_period_us (Permission denied)
> org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler.updateCgroup(CgroupsLCEResourcesHandler.java:263)
> This JIRA is about automatically creating YARN control group in the case 
> above. It reduces the cost of administration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5739) Provide timeline reader API to list available timeline entity types for one application

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691199#comment-15691199
 ] 

Varun Saxena commented on YARN-5739:


I think this patch can be committed before YARN-5585 as its available.

Few nits though.
# In javadoc for TimelineReader#getEntityTypes. Instead of below, it should be 
"A set which contains available timeline entity types, represented as strings 
if found, empty otherwise.". Also the line in front of throws should say "while 
getting entity types" instead of "while getting entity".
{code}
190* @return A set contains available timeine entity types, represented 
as
191* Strings if found, null otherwise.
192* @throws IOException  if any problem occurs while getting entity.
{code}
# The javadoc above EntityTypeReader#getNextRowKey should have application id 
instead of application information
{code}
137* @param currRowKeyPrefix The current prefix that contains user, 
cluster,
138* flow, run, and application information.
{code}
# TimelineReaderManager lines 197 and 198 can be combined into one line.

I am +1 on the latest patch once this is fixed and will commit it by tomorrow 
end of day.
[~sjlee0], you can probably go through the patch as well.

Do let me know if you want me to hold off the commit due to Thanksgiving.

> Provide timeline reader API to list available timeline entity types for one 
> application
> ---
>
> Key: YARN-5739
> URL: https://issues.apache.org/jira/browse/YARN-5739
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Li Lu
>Assignee: Li Lu
> Attachments: YARN-5739-YARN-5355.001.patch, 
> YARN-5739-YARN-5355.002.patch, YARN-5739-YARN-5355.003.patch, 
> YARN-5739-YARN-5355.004.patch
>
>
> Right now we only show a part of available timeline entity data in the new 
> YARN UI. However, some data (especially library specific data) are not 
> possible to be queried out by the web UI. It will be appealing for the UI to 
> provide an "entity browser" for each YARN application. Actually, simply 
> dumping out available timeline entities (with proper pagination, of course) 
> would be pretty helpful for UI users. 
> On timeline side, we're not far away from this goal. Right now I believe the 
> only thing missing is to list all available entity types within one 
> application. The challenge here is that we're not storing this data for each 
> application, but given this kind of call is relatively rare (compare to 
> writes and updates) we can perform some scanning during the read time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5725) Test uncaught exception in TestContainersMonitorResourceChange.testContainersResourceChange when setting IP and host

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691162#comment-15691162
 ] 

Daniel Templeton commented on YARN-5725:


Thanks, [~miklos.szeg...@cloudera.com], for the updated patch.  One last, tiny 
nit.  There two lines: {code}Mockito.doReturn(containerMap)
 .when(context).getContainers();{code} can be combined.

> Test uncaught exception in 
> TestContainersMonitorResourceChange.testContainersResourceChange when setting 
> IP and host
> 
>
> Key: YARN-5725
> URL: https://issues.apache.org/jira/browse/YARN-5725
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
>  Labels: oct16-easy
> Attachments: YARN-5725.000.patch, YARN-5725.001.patch, 
> YARN-5725.002.patch, YARN-5725.003.patch, YARN-5725.004.patch, 
> YARN-5725.005.patch, YARN-5725.006.patch, YARN-5725.007.patch, 
> YARN-5725.008.patch, YARN-5725.009.patch, YARN-5725.010.patch
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> The issue is a warning but it prevents container monitor to continue
> 2016-10-12 14:38:23,280 WARN  [Container Monitor] 
> monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(594)) - 
> Uncaught exception in ContainersMonitorImpl while monitoring resource of 
> container_123456_0001_01_01
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:455)
> 2016-10-12 14:38:23,281 WARN  [Container Monitor] 
> monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(613)) - 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>  is interrupted. Exiting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5921) Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691114#comment-15691114
 ] 

Varun Saxena commented on YARN-5921:


TestRMRestart test failure to be tracked by YARN-5548

> Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState
> --
>
> Key: YARN-5921
> URL: https://issues.apache.org/jira/browse/YARN-5921
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-5921.01.patch, YARN-5921.02.patch
>
>
> Code in RMContextImpl is as under:
> {code:title=RMContextImpl.java|borderStyle=solid}
>   void setHAServiceState(HAServiceState haServiceState) {
> synchronized (haServiceState) {
>   this.haServiceState = haServiceState;
> }
>   }
>   public HAServiceState getHAServiceState() {
> synchronized (haServiceState) {
>   return haServiceState;
> }
>   }
> {code}
> As can be seen above, in setHAServiceState, we are synchronizing on the 
> passed haServiceState instead of haServiceState in RMContextImpl which will 
> not lead to desired effect. This does not seem to be intentional.
> We can use a RW lock or synchronize on some object here. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5920) Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5920:
---
Summary: Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang  
(was: TestRMHA.testTransitionedToStandbyShouldNotHang is flaky)

> Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang
> ---
>
> Key: YARN-5920
> URL: https://issues.apache.org/jira/browse/YARN-5920
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Rohith Sharma K S
>Assignee: Varun Saxena
> Attachments: ThreadDump.txt, YARN-5920.01.patch, YARN-5920.02.patch
>
>
> In build 
> [linkg|https://builds.apache.org/job/PreCommit-YARN-Build/13986/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt]
>  test case timed out. This need to be investigated.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5903) Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl beforeclass setup method

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691102#comment-15691102
 ] 

Varun Saxena commented on YARN-5903:


[~haibochen], previously we were throwing IOException if RM did not change to 
STARTED state.
So shouldn't we check result of CountDownLatch#await and if its false, throw an 
IOException  as that will indicate RM has not yet started.

> Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl 
> beforeclass setup method
> 
>
> Key: YARN-5903
> URL: https://issues.apache.org/jira/browse/YARN-5903
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: yarn5903.001.patch
>
>
> This is essentially the same race condition as in YARN-5901, that is, 
> resourcemanager.getServiceState() == STATE.STARTED does not guarantee 
> resource manager is fully started.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5649) Add REST endpoints for updating application timeouts

[Rohith Sharma K S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-5649:

Attachment: YARN-5649.3.patch

Updating the patch adding new GET api for collections. Rest of all remains 
same. For PUT, /timeouts is not implemented now. 

> Add REST endpoints for updating application timeouts
> 
>
> Key: YARN-5649
> URL: https://issues.apache.org/jira/browse/YARN-5649
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: YARN-5649.1.patch, YARN-5649.2.patch, YARN-5649.3.patch, 
> YARN-5649.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5903) Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl beforeclass setup method

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691063#comment-15691063
 ] 

Varun Saxena commented on YARN-5903:


bq. Yes, we have seen consistent failures on some of our machines. My guess is 
the thread that starts the resource manager is always delayed on that platform. 
It is delayed so much so that client cannot reach the sever even with 10 
retries.
Okay...The fix makes sense. Let me have a closer look.

> Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl 
> beforeclass setup method
> 
>
> Key: YARN-5903
> URL: https://issues.apache.org/jira/browse/YARN-5903
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: yarn5903.001.patch
>
>
> This is essentially the same race condition as in YARN-5901, that is, 
> resourcemanager.getServiceState() == STATE.STARTED does not guarantee 
> resource manager is fully started.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5901) Fix race condition in TestGetGroups beforeclass setup()

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691058#comment-15691058
 ] 

Daniel Templeton commented on YARN-5901:


Ditto on the error message.  You can assert that the {{await()}} is true.  In 
fact, without the assert, you'll continue on even if the RM doesn't start.

Otherwise, looks great.  Nice use of a {{CountDownLatch}}.

> Fix race condition in TestGetGroups beforeclass setup()
> ---
>
> Key: YARN-5901
> URL: https://issues.apache.org/jira/browse/YARN-5901
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>  Labels: unittest
> Attachments: yarn5901.001.patch
>
>
> In TestGetGroups, the class-level setup method spins up, in a child thread, a 
> resource manager that Yarn clients can talk to. But it checks whether the 
> resource manager is fully started by doing resourcemanager.getServiceState() 
> == STATE.STARTED. This is not reliable since resourcemanager.start() will 
> first trigger service state change in RM, and then starts up all the services 
> added to RM. We need to wait for RM to fully start before YARN clients  can 
> send requests. Otherwise, the tests can fail due to "connection refused"  
> exception when the main thread sends out client requests to RM and if the RPC 
> server has not fired up in the child thread.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5924) Resource Manager fails to load state with InvalidProtocolBufferException

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691051#comment-15691051
 ] 

Hadoop QA commented on YARN-5924:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 21s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 6 new + 43 unchanged - 0 fixed = 49 total (was 43) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 44m 
52s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 48s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5924 |
| GITHUB PR | https://github.com/apache/hadoop/pull/164 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux e50687820f03 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 005850b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/14064/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-YARN-Build/14064/artifact/patchprocess/whitespace-eol.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14064/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14064/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Resource 

[jira] [Commented] (YARN-3409) Add constraint node labels

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691048#comment-15691048
 ] 

Naganarasimha G R commented on YARN-3409:
-


Thanks [~bibinchundatt], [~varun_saxena] & [~rohithsharma], for taking the 
discussion ahead.
bq. We should call current feature as NodeConstraints since we have label is 
considered as cluster partition.Resource request as constraintsExpression 
instead of constraintslabelexpression.
Actually how the way it has progressed is like there are 2 kinds of labels, 
{{"Partition"}} where in its used to segregate the cluster & as well gurantee 
the capacities, and other is {{"Constraints"}} which tries to capture the 
attribute of the node. Agree that even if it sounds similar its different in 
functionality but based on community feedback we can further decide on the 
naming convention.

bq. Nodes with same Constraints could be in multiple partition,so are we 
allowing resource to serve any labels satisfying constraints or only subset of 
label resource pool satisfying constraints.
As Varun mentioned, it would be like for the nodes satisfying 
partition(expression, multiple in case of ||), constraints expression would be 
applied to further filter and do the allocation. 

bq. At RM side is it required to have a constraints list?.We could consider all 
constraints or attributes with which NM registers.For dynamic updation its 
better that way rt?
As we have type for each constraint, we need to ensure that two NMs's dont give 
constaints with same name but of different type. Hence we need to have a 
superset in RM based on which evaluations can be done. Plans are there to 
dynamically add the constraints and then register the NM-constraint mapping in 
case the constriant doesnt exist.

bq. Any relaxation similar to node locality.If request doesnt satisfy 
constraints will it ever serve request.If we are planning to support then 
addition attribute in ResourceRequest.
Well currently we plan to make it strict enforcement for constraints and 
relaxation, even if req we can have it when YARN-4902 supports 
{code}
Delayed-OR [
- Give me default partition with constraint expr1, 
(Wait for x mins)
- Any node in default partition. 
]
{code}

bq. currently headroom is calculated per partition basis, how does head room 
will be calculated once we support constraint label? What is the impact ? Does 
it application wait for ever if required constraint label not found?
Yes as Varun mentioned Constraints will merely be used for node selection and 
neither are capacities guranteed for it, and hence headroom will not be 
considered. Headroom we can provide for a partition per application here it can 
be an expression so it cannot be guaranteeing definite heardroom for 
application as there can overlap of constraints across the resource requests.
As per current plan application will wait for ever if required constaint is not 
available.

bq. So for constraint based Resource Request will it be always of type ANY . 
Schedulers have node locality delay during allocation .Does node locality 
matter for resource request with constraint label?
Yes similar to labels we initially plan to consider the constraint expression 
present in ANY as final expression. As mentioned earlier nodes with matching 
locality and constraints will be tried to picked if failed then nodes in the 
same rack (of the specified nodes) satifying the constriant expression will be 
picked else any nodes satisfying the constraint expression will be picked.

bq. Let us create a new branch for this work.
Yes [~varun_saxena] as per initial discussion with [~wangda], he also suggested 
the same as the changes might not fit into smaller number of patches.

> Add constraint node labels
> --
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, capacityscheduler, client
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
> Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to 
> determinate how resources of a special set of nodes could be shared by a 
> group of entities (like teams, departments, etc.). Partitions of a cluster 
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has 
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% 
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of 
> node’s hardware/software just for affinity. Some example of constraints:
> - glibc version
> - 

[jira] [Commented] (YARN-5921) Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691016#comment-15691016
 ] 

Hadoop QA commented on YARN-5921:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 0 new + 11 unchanged - 1 fixed = 11 total (was 12) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 18s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 26s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5921 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840278/YARN-5921.02.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 63879f9fb4d3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 005850b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/14062/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14062/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14062/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |

[jira] [Commented] (YARN-5877) Allow all nm-whitelist-env to get overridden during launch

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691010#comment-15691010
 ] 

Hadoop QA commented on YARN-5877:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
26s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m  
2s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 53s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 3 new + 208 unchanged - 1 fixed = 211 total (was 209) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 38s{color} 
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 
48s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
37s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 21s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5877 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840288/YARN-5877.0003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 22dfd93974ee 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 3541ed8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/14061/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
 |
| unit | 

[jira] [Commented] (YARN-5890) FairScheduler should log information about AM-resource-usage and max-AM-share for queues

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690997#comment-15690997
 ] 

Daniel Templeton commented on YARN-5890:


Thanks, [~yufeigu].  Minor points on the tests:

* {{queueFSNoneZero}} should be {{queueFSNonZero}}.
* Since max AM share isn't being set explicitly, it might be a good idea to add 
a comment stating what it you expect the default to be.  It doesn't really 
affect anything, but as a reader of the code, it's the first question that 
comes to mind.
* I'd love to see some messages on all those asserts.
* I know that none of the methods here change {{amResource}}, but since 
{{Resource}} is mutable, I'd feel better if we created a new one for each part 
of the test rather than reusing the same {{amResource}}.

> FairScheduler should log information about AM-resource-usage and max-AM-share 
> for queues
> 
>
> Key: YARN-5890
> URL: https://issues.apache.org/jira/browse/YARN-5890
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5890.001.patch, YARN-5890.002.patch
>
>
> There are several cases where jobs in a queue or stuck likely because of 
> maxAMShare. It is hard to debug these issues without any information.
> At the very least, we need to log both AM-resource-usage and max-AM-share for 
> queues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5774) MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set yarn.scheduler.minimum-allocation-mb to 0.

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690959#comment-15690959
 ] 

Yufei Gu commented on YARN-5774:


[~templedf], sure, and thanks for the review.

> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set 
> yarn.scheduler.minimum-allocation-mb to 0.
> 
>
> Key: YARN-5774
> URL: https://issues.apache.org/jira/browse/YARN-5774
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>  Labels: oct16-easy
> Attachments: YARN-5774.001.patch, YARN-5774.002.patch, 
> YARN-5774.003.patch, YARN-5774.004.patch, YARN-5774.005.patch, 
> YARN-5774.006.patch
>
>
> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler 
> because there is no resource request for the AM. This happened when you 
> configure {{yarn.scheduler.minimum-allocation-mb}} to zero.
> The problem is in the code used by both Capacity Scheduler and Fair 
> Scheduler. {{scheduler.increment-allocation-mb}} is a concept in FS, but not 
> CS. So the common code in class RMAppManager passes the 
> {{yarn.scheduler.minimum-allocation-mb}} as incremental one because there is 
> no incremental one for CS when it tried to normalize the resource requests.
> {code}
>  SchedulerUtils.normalizeRequest(amReq, scheduler.getResourceCalculator(),
>   scheduler.getClusterResource(),
>   scheduler.getMinimumResourceCapability(),
>   scheduler.getMaximumResourceCapability(),
>   scheduler.getMinimumResourceCapability());  --> incrementResource 
> should be passed here.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5572) HBaseTimelineWriterImpl appears to reference a bad property name

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690956#comment-15690956
 ] 

Hadoop QA commented on YARN-5572:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
48s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
45s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in 
the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 24s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5572 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840099/YARN-5572.01.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 7ffd7418aad9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 005850b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14063/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
 U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14063/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was 

[jira] [Commented] (YARN-5918) Handle Opportunistic scheduling allocate request failure when NM is lost

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690948#comment-15690948
 ] 

Hudson commented on YARN-5918:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10880 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10880/])
YARN-5918. Handle Opportunistic scheduling allocate request failure when (arun 
suresh: rev 005850b28feb2f7bb8c2844d11e3f9d21b45d754)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/distributed/NodeQueueLoadMonitor.java


> Handle Opportunistic scheduling allocate request failure when NM is lost
> 
>
> Key: YARN-5918
> URL: https://issues.apache.org/jira/browse/YARN-5918
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-5918.0001.patch, YARN-5918.0002.patch, 
> YARN-5918.0003.patch, YARN-5918.0004.patch
>
>
> Allocate request failure during Opportunistic container allocation when 
> nodemanager is lost 
> {noformat}
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root 
> OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS  
> APPID=application_1479637990302_0002
> CONTAINERID=container_e12_1479637990302_0002_01_06  
> RESOURCE=
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Removed node docker2:38297 clusterResource: 
> 2016-11-20 10:38:49,434 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 7 on 8030, call Call#35 Retry#0 
> org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 
> 172.17.0.2:51584
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNode(OpportunisticContainerAllocatorAMService.java:420)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNodes(OpportunisticContainerAllocatorAMService.java:412)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.getLeastLoadedNodes(OpportunisticContainerAllocatorAMService.java:402)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.allocate(OpportunisticContainerAllocatorAMService.java:236)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:846)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:789)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2539)
> 2016-11-20 10:38:50,824 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> container_e12_1479637990302_0002_01_02 Container Transitioned from 
> RUNNING to COMPLETED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5918) Handle Opportunistic scheduling allocate request failure when NM is lost

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5918:
--
Summary: Handle Opportunistic scheduling allocate request failure when NM 
is lost  (was: Opportunistic scheduling allocate request failure when NM lost)

> Handle Opportunistic scheduling allocate request failure when NM is lost
> 
>
> Key: YARN-5918
> URL: https://issues.apache.org/jira/browse/YARN-5918
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-5918.0001.patch, YARN-5918.0002.patch, 
> YARN-5918.0003.patch, YARN-5918.0004.patch
>
>
> Allocate request failure during Opportunistic container allocation when 
> nodemanager is lost 
> {noformat}
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root 
> OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS  
> APPID=application_1479637990302_0002
> CONTAINERID=container_e12_1479637990302_0002_01_06  
> RESOURCE=
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Removed node docker2:38297 clusterResource: 
> 2016-11-20 10:38:49,434 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 7 on 8030, call Call#35 Retry#0 
> org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 
> 172.17.0.2:51584
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNode(OpportunisticContainerAllocatorAMService.java:420)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNodes(OpportunisticContainerAllocatorAMService.java:412)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.getLeastLoadedNodes(OpportunisticContainerAllocatorAMService.java:402)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.allocate(OpportunisticContainerAllocatorAMService.java:236)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:846)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:789)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2539)
> 2016-11-20 10:38:50,824 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> container_e12_1479637990302_0002_01_02 Container Transitioned from 
> RUNNING to COMPLETED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5877) Allow all nm-whitelist-env to get overridden during launch

[Bibin A Chundatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bibin A Chundatt updated YARN-5877:
---
Attachment: YARN-5877.0003.patch

[~sunilg]. Attaching patch based on separate parameter. Only when container env 
is of type docker. Hope this patch is inline with your suggestion of docker 
specific too.

> Allow all nm-whitelist-env to get overridden during launch
> --
>
> Key: YARN-5877
> URL: https://issues.apache.org/jira/browse/YARN-5877
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: Dockerfile, YARN-5877.0001.patch, YARN-5877.0002.patch, 
> YARN-5877.0003.patch, bootstrap.sh, yarn-site.xml
>
>
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values 
> should  containers may override rather than use NodeManager's default.
> {code}
>   
> Environment variables that containers may override rather 
> than use NodeManager's default.
> yarn.nodemanager.env-whitelist
> 
> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME
>   
> {code}
> But only the following containers can override
> {code}
> whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
> whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5918) Opportunistic scheduling allocate request failure when NM lost

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5918:
--
Fix Version/s: 3.0.0-alpha2

> Opportunistic scheduling allocate request failure when NM lost
> --
>
> Key: YARN-5918
> URL: https://issues.apache.org/jira/browse/YARN-5918
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-5918.0001.patch, YARN-5918.0002.patch, 
> YARN-5918.0003.patch, YARN-5918.0004.patch
>
>
> Allocate request failure during Opportunistic container allocation when 
> nodemanager is lost 
> {noformat}
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root 
> OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS  
> APPID=application_1479637990302_0002
> CONTAINERID=container_e12_1479637990302_0002_01_06  
> RESOURCE=
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Removed node docker2:38297 clusterResource: 
> 2016-11-20 10:38:49,434 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 7 on 8030, call Call#35 Retry#0 
> org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 
> 172.17.0.2:51584
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNode(OpportunisticContainerAllocatorAMService.java:420)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNodes(OpportunisticContainerAllocatorAMService.java:412)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.getLeastLoadedNodes(OpportunisticContainerAllocatorAMService.java:402)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.allocate(OpportunisticContainerAllocatorAMService.java:236)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:846)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:789)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2539)
> 2016-11-20 10:38:50,824 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> container_e12_1479637990302_0002_01_02 Container Transitioned from 
> RUNNING to COMPLETED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5921) Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690872#comment-15690872
 ] 

Daniel Templeton commented on YARN-5921:


+1 on the latest patch

> Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState
> --
>
> Key: YARN-5921
> URL: https://issues.apache.org/jira/browse/YARN-5921
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-5921.01.patch, YARN-5921.02.patch
>
>
> Code in RMContextImpl is as under:
> {code:title=RMContextImpl.java|borderStyle=solid}
>   void setHAServiceState(HAServiceState haServiceState) {
> synchronized (haServiceState) {
>   this.haServiceState = haServiceState;
> }
>   }
>   public HAServiceState getHAServiceState() {
> synchronized (haServiceState) {
>   return haServiceState;
> }
>   }
> {code}
> As can be seen above, in setHAServiceState, we are synchronizing on the 
> passed haServiceState instead of haServiceState in RMContextImpl which will 
> not lead to desired effect. This does not seem to be intentional.
> We can use a RW lock or synchronize on some object here. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5761) Separate QueueManager from Scheduler

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690870#comment-15690870
 ] 

Sunil G commented on YARN-5761:
---

Thanks [~xgong]. I think there are some java doc issues. Could u pls help to 
update the same if you have bandwidth. 

> Separate QueueManager from Scheduler
> 
>
> Key: YARN-5761
> URL: https://issues.apache.org/jira/browse/YARN-5761
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>  Labels: oct16-medium
> Attachments: YARN-5761.1.patch, YARN-5761.1.rebase.patch, 
> YARN-5761.2.patch, YARN-5761.3.patch, YARN-5761.4.patch, YARN-5761.5.patch, 
> YARN-5761.6.patch, YARN-5761.7.patch, YARN-5761.7.patch
>
>
> Currently, in scheduler code, we are doing queue manager and scheduling work. 
> We'd better separate the queue manager out of scheduler logic. In that case, 
> it would be much easier and safer to extend.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5865) Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690737#comment-15690737
 ] 

Rohith Sharma K S commented on YARN-5865:
-

[~sunilg] Is test cases are related? Could you check it please?

> Retrospect updateApplicationPriority api to handle state store exception in 
> align with YARN-5611
> 
>
> Key: YARN-5865
> URL: https://issues.apache.org/jira/browse/YARN-5865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5865.0001.patch, YARN-5865.0002.patch, 
> YARN-5865.0003.patch, YARN-5865.0004.patch, YARN-5865.0005.patch, 
> YARN-5865.0006.patch, YARN-5865.branch-2.0001.patch, 
> YARN-5865.branch-2.0002.patch
>
>
> Post YARN-5611, revisit dynamic update of application priority logic with 
> respect to state store error handling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5865) Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690757#comment-15690757
 ] 

Sunil G commented on YARN-5865:
---

|YARN-5816|TestDelegationTokenRenewer.testCancelWithMultipleAppSubmissions|
|YARN-5548|TestRMRestart.testFinishedAppRemovalAfterRMRestart|
|YARN-5343|TestContinuousScheduling.testSortedNodes|

Last one is failed even after the fix in YARN-5343, mostly needs to reopen. So 
test failures are not related.

> Retrospect updateApplicationPriority api to handle state store exception in 
> align with YARN-5611
> 
>
> Key: YARN-5865
> URL: https://issues.apache.org/jira/browse/YARN-5865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5865.0001.patch, YARN-5865.0002.patch, 
> YARN-5865.0003.patch, YARN-5865.0004.patch, YARN-5865.0005.patch, 
> YARN-5865.0006.patch, YARN-5865.branch-2.0001.patch, 
> YARN-5865.branch-2.0002.patch
>
>
> Post YARN-5611, revisit dynamic update of application priority logic with 
> respect to state store error handling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5918) Opportunistic scheduling allocate request failure when NM lost

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690702#comment-15690702
 ] 

Hadoop QA commented on YARN-5918:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 42m 
13s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 42s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5918 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840271/YARN-5918.0004.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux d5e558a8c1ae 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 3541ed8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14054/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14054/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Opportunistic scheduling allocate request failure when NM lost
> --
>
> Key: YARN-5918
> URL: https://issues.apache.org/jira/browse/YARN-5918
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-5918.0001.patch, YARN-5918.0002.patch, 
> YARN-5918.0003.patch, YARN-5918.0004.patch
>
>
> Allocate request 

[jira] [Updated] (YARN-5921) Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5921:
---
Attachment: YARN-5921.02.patch

> Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState
> --
>
> Key: YARN-5921
> URL: https://issues.apache.org/jira/browse/YARN-5921
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-5921.01.patch, YARN-5921.02.patch
>
>
> Code in RMContextImpl is as under:
> {code:title=RMContextImpl.java|borderStyle=solid}
>   void setHAServiceState(HAServiceState haServiceState) {
> synchronized (haServiceState) {
>   this.haServiceState = haServiceState;
> }
>   }
>   public HAServiceState getHAServiceState() {
> synchronized (haServiceState) {
>   return haServiceState;
> }
>   }
> {code}
> As can be seen above, in setHAServiceState, we are synchronizing on the 
> passed haServiceState instead of haServiceState in RMContextImpl which will 
> not lead to desired effect. This does not seem to be intentional.
> We can use a RW lock or synchronize on some object here. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5921) Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690644#comment-15690644
 ] 

Varun Saxena commented on YARN-5921:


Thanks [~templedf] for the review.
You are correct. Javadoc of ReentrantReadWriteLock suggests the same as well. 

Will change to synchronizing on an object.

> Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState
> --
>
> Key: YARN-5921
> URL: https://issues.apache.org/jira/browse/YARN-5921
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-5921.01.patch
>
>
> Code in RMContextImpl is as under:
> {code:title=RMContextImpl.java|borderStyle=solid}
>   void setHAServiceState(HAServiceState haServiceState) {
> synchronized (haServiceState) {
>   this.haServiceState = haServiceState;
> }
>   }
>   public HAServiceState getHAServiceState() {
> synchronized (haServiceState) {
>   return haServiceState;
> }
>   }
> {code}
> As can be seen above, in setHAServiceState, we are synchronizing on the 
> passed haServiceState instead of haServiceState in RMContextImpl which will 
> not lead to desired effect. This does not seem to be intentional.
> We can use a RW lock or synchronize on some object here. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5762) Summarize ApplicationNotFoundException in the RM log

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690570#comment-15690570
 ] 

Daniel Templeton commented on YARN-5762:


Yeah, the number of applications could be quite large on some clusters.

> Summarize ApplicationNotFoundException in the RM log
> 
>
> Key: YARN-5762
> URL: https://issues.apache.org/jira/browse/YARN-5762
> Project: Hadoop YARN
>  Issue Type: Task
>Affects Versions: 2.7.2
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
>Priority: Minor
> Attachments: YARN-5762.01.patch
>
>
> We found a lot of {{ApplicationNotFoundException}} in the RM logs. These were 
> most likely caused by the {{AggregatedLogDeletionService}} [which 
> checks|https://github.com/apache/hadoop/blob/262827cf75bf9c48cd95335eb04fd8ff1d64c538/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogDeletionService.java#L156]
>  that the application is not running anymore. e.g.
> {code}2016-10-17 15:25:26,542 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 20 on 8032, call 
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport 
> from :12205 Call#35401 Retry#0
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application 
> with id 'application_1473396553140_1451' doesn't exist in RM.
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:327)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:175)
> at 
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:417)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> 2016-10-17 15:25:26,633 INFO org.apache.hadoop.ipc.Server: IPC Server handler 
> 47 on 8032, call 
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport 
> from :12205 Call#35404 Retry#0
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application 
> with id 'application_1473396553140_1452' doesn't exist in RM.
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:327)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:175)
> at 
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:417)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5865) Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690548#comment-15690548
 ] 

Hadoop QA commented on YARN-5865:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 8 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
35s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
34s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
38s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 29s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 2 new + 643 unchanged - 2 fixed = 645 total (was 645) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 15s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK 
v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 39s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_111 Failed junit tests | 
hadoop.yarn.server.resourcemanager.TestRMRestart |
| JDK v1.7.0_111 Failed junit tests | 
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
|   | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | YARN-5865 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840250/YARN-5865.branch-2.0002.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  

[jira] [Commented] (YARN-4206) Add life time value in Application report and CLI

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690515#comment-15690515
 ] 

Hadoop QA commented on YARN-4206:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
49s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  5m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m  
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 50s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 11 new + 374 unchanged - 2 fixed = 385 total (was 376) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m 
42s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 
17s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 40s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-4206 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840247/YARN-4206.5.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  cc  |
| uname | Linux 831d89e5b8d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 3541ed8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 

[jira] [Updated] (YARN-5918) Opportunistic scheduling allocate request failure when NM lost

[Bibin A Chundatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bibin A Chundatt updated YARN-5918:
---
Attachment: YARN-5918.0004.patch

Attaching patch after moving testcase

> Opportunistic scheduling allocate request failure when NM lost
> --
>
> Key: YARN-5918
> URL: https://issues.apache.org/jira/browse/YARN-5918
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-5918.0001.patch, YARN-5918.0002.patch, 
> YARN-5918.0003.patch, YARN-5918.0004.patch
>
>
> Allocate request failure during Opportunistic container allocation when 
> nodemanager is lost 
> {noformat}
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root 
> OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS  
> APPID=application_1479637990302_0002
> CONTAINERID=container_e12_1479637990302_0002_01_06  
> RESOURCE=
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Removed node docker2:38297 clusterResource: 
> 2016-11-20 10:38:49,434 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 7 on 8030, call Call#35 Retry#0 
> org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 
> 172.17.0.2:51584
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNode(OpportunisticContainerAllocatorAMService.java:420)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNodes(OpportunisticContainerAllocatorAMService.java:412)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.getLeastLoadedNodes(OpportunisticContainerAllocatorAMService.java:402)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.allocate(OpportunisticContainerAllocatorAMService.java:236)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:846)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:789)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2539)
> 2016-11-20 10:38:50,824 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> container_e12_1479637990302_0002_01_02 Container Transitioned from 
> RUNNING to COMPLETED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5795) FairScheduler set AppMaster vcores didn't work

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690456#comment-15690456
 ] 

Daniel Templeton commented on YARN-5795:


[~yufeigu] is in the process of overhauling the resource normalization process 
in YARN-5774.  At a minimum, we should wait until that change is in.  It would 
be nice, though, to also fix this issue in that patch.

> FairScheduler set AppMaster vcores didn't work
> --
>
> Key: YARN-5795
> URL: https://issues.apache.org/jira/browse/YARN-5795
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: tangshangwen
> Attachments: 0001-YARN-5795.patch
>
>
> In our cluster, we use the linux container, I would like to increase the 
> number of cpu to get more CPU time slice, but it did not take effect， i set 
> yarn.app.mapreduce.am.resource.cpu-vcores = 2 ,but i found the 
> resourcemanager log
> {noformat}
> [2016-10-27T16:36:37.280 08:00] [INFO] 
> resourcemanager.scheduler.SchedulerNode.allocateContainer(SchedulerNode.java 
> 153) [ResourceManager Event Processor] : Assigned container 
> container_1477059529836_336635_01_01 of capacity 
> {noformat}
> Because scheduler.getResourceCalculator() only computes memory
> {code:title=RMAppManager.java|borderStyle=solid}
>  private ResourceRequest validateAndCreateResourceRequest(
>   ApplicationSubmissionContext submissionContext, boolean isRecovery)
>   throws InvalidResourceRequestException {
> ...
>   SchedulerUtils.normalizeRequest(amReq, 
> scheduler.getResourceCalculator(),
>   scheduler.getClusterResource(),
>   scheduler.getMinimumResourceCapability(),
>   scheduler.getMaximumResourceCapability(),
>   scheduler.getMinimumResourceCapability());
> ...
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5774) MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set yarn.scheduler.minimum-allocation-mb to 0.

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690460#comment-15690460
 ] 

Daniel Templeton commented on YARN-5774:


[~yufeigu], can you take a look at YARN-5795 and see 1) what you think of it, 
and 2) if that's an issue you can also tackle in this patch?  Thanks!

> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set 
> yarn.scheduler.minimum-allocation-mb to 0.
> 
>
> Key: YARN-5774
> URL: https://issues.apache.org/jira/browse/YARN-5774
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>  Labels: oct16-easy
> Attachments: YARN-5774.001.patch, YARN-5774.002.patch, 
> YARN-5774.003.patch, YARN-5774.004.patch, YARN-5774.005.patch, 
> YARN-5774.006.patch
>
>
> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler 
> because there is no resource request for the AM. This happened when you 
> configure {{yarn.scheduler.minimum-allocation-mb}} to zero.
> The problem is in the code used by both Capacity Scheduler and Fair 
> Scheduler. {{scheduler.increment-allocation-mb}} is a concept in FS, but not 
> CS. So the common code in class RMAppManager passes the 
> {{yarn.scheduler.minimum-allocation-mb}} as incremental one because there is 
> no incremental one for CS when it tried to normalize the resource requests.
> {code}
>  SchedulerUtils.normalizeRequest(amReq, scheduler.getResourceCalculator(),
>   scheduler.getClusterResource(),
>   scheduler.getMinimumResourceCapability(),
>   scheduler.getMaximumResourceCapability(),
>   scheduler.getMinimumResourceCapability());  --> incrementResource 
> should be passed here.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5918) Opportunistic scheduling allocate request failure when NM lost

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690316#comment-15690316
 ] 

Hadoop QA commented on YARN-5918:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m  2s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 26s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5918 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840245/YARN-5918.0003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux d1a436e4dc80 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 3541ed8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/14051/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14051/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14051/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Opportunistic scheduling allocate request failure when NM lost
> --
>
> Key: 

[jira] [Commented] (YARN-5918) Opportunistic scheduling allocate request failure when NM lost

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690315#comment-15690315
 ] 

Arun Suresh commented on YARN-5918:
---

Thanks for the patch and the testcase [~bibinchundatt].

We already have a 
{{o.a.h.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService}}.
 Lets move the new test into that class.

+1 from me pending jenkins

> Opportunistic scheduling allocate request failure when NM lost
> --
>
> Key: YARN-5918
> URL: https://issues.apache.org/jira/browse/YARN-5918
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-5918.0001.patch, YARN-5918.0002.patch, 
> YARN-5918.0003.patch
>
>
> Allocate request failure during Opportunistic container allocation when 
> nodemanager is lost 
> {noformat}
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root 
> OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS  
> APPID=application_1479637990302_0002
> CONTAINERID=container_e12_1479637990302_0002_01_06  
> RESOURCE=
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Removed node docker2:38297 clusterResource: 
> 2016-11-20 10:38:49,434 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 7 on 8030, call Call#35 Retry#0 
> org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 
> 172.17.0.2:51584
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNode(OpportunisticContainerAllocatorAMService.java:420)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNodes(OpportunisticContainerAllocatorAMService.java:412)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.getLeastLoadedNodes(OpportunisticContainerAllocatorAMService.java:402)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.allocate(OpportunisticContainerAllocatorAMService.java:236)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:846)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:789)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2539)
> 2016-11-20 10:38:50,824 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> container_e12_1479637990302_0002_01_02 Container Transitioned from 
> RUNNING to COMPLETED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5694) ZKRMStateStore can prevent the transition to standby in branch-2.7 if the ZK node is unreachable

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690309#comment-15690309
 ] 

Daniel Templeton commented on YARN-5694:


The test failures appear to be HADOOP-12687 and YARN-2634.  I'm doing some 
additional testing to make sure the {{TestClientRMTokens}} failures really are 
unrelated.  I should know later today.

> ZKRMStateStore can prevent the transition to standby in branch-2.7 if the ZK 
> node is unreachable
> 
>
> Key: YARN-5694
> URL: https://issues.apache.org/jira/browse/YARN-5694
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.3
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>  Labels: oct16-medium
> Attachments: YARN-5694.001.patch, YARN-5694.002.patch, 
> YARN-5694.003.patch, YARN-5694.004.patch, YARN-5694.004.patch, 
> YARN-5694.005.patch, YARN-5694.006.patch, YARN-5694.007.patch, 
> YARN-5694.008.patch, YARN-5694.branch-2.7.001.patch, 
> YARN-5694.branch-2.7.002.patch, YARN-5694.branch-2.7.004.patch, 
> YARN-5694.branch-2.7.005.patch
>
>
> {{ZKRMStateStore.doStoreMultiWithRetries()}} holds the lock while trying to 
> talk to ZK.  If the connection fails, it will retry while still holding the 
> lock.  The retries are intended to be strictly time limited, but in the case 
> that the ZK node is unreachable, the time limit fails, resulting in the 
> thread holding the lock for over an hour.  Transitioning the RM to standby 
> requires that same lock, so in exactly the case that the RM should be 
> transitioning to standby, the {{VerifyActiveStatusThread}} blocks it from 
> happening.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2634) Test failure for TestClientRMTokens

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690298#comment-15690298
 ] 

Daniel Templeton commented on YARN-2634:


I'm still seeing this issue on branch-2.7 
(https://builds.apache.org/job/PreCommit-YARN-Build/14035/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt):

{noformat}
Tests run: 6, Failures: 3, Errors: 2, Skipped: 0, Time elapsed: 17.669 sec <<< 
FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
testShortCircuitRenewCancelDifferentHostSamePort(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
  Time elapsed: 0.316 sec  <<< FAILURE!
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.checkShortCircuitRenewCancel(TestClientRMTokens.java:363)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testShortCircuitRenewCancelDifferentHostSamePort(TestClientRMTokens.java:316)

testShortCircuitRenewCancelDifferentHostDifferentPort(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
  Time elapsed: 0.068 sec  <<< FAILURE!
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.checkShortCircuitRenewCancel(TestClientRMTokens.java:363)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testShortCircuitRenewCancelDifferentHostDifferentPort(TestClientRMTokens.java:327)

testShortCircuitRenewCancel(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
  Time elapsed: 0.058 sec  <<< ERROR!
java.lang.NullPointerException: null
at 
org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier$Renewer.getRmClient(RMDelegationTokenIdentifier.java:144)
at 
org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier$Renewer.renew(RMDelegationTokenIdentifier.java:97)
at org.apache.hadoop.security.token.Token.renew(Token.java:377)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.checkShortCircuitRenewCancel(TestClientRMTokens.java:353)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testShortCircuitRenewCancel(TestClientRMTokens.java:285)

testShortCircuitRenewCancelSameHostDifferentPort(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
  Time elapsed: 0.045 sec  <<< FAILURE!
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.checkShortCircuitRenewCancel(TestClientRMTokens.java:363)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testShortCircuitRenewCancelSameHostDifferentPort(TestClientRMTokens.java:305)

testShortCircuitRenewCancelWildcardAddress(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
  Time elapsed: 0.033 sec  <<< ERROR!
java.lang.NullPointerException: null
at org.apache.hadoop.net.NetUtils.isLocalAddress(NetUtils.java:686)
at 
org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier$Renewer.getRmClient(RMDelegationTokenIdentifier.java:145)
at 
org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier$Renewer.renew(RMDelegationTokenIdentifier.java:97)
at org.apache.hadoop.security.token.Token.renew(Token.java:377)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.checkShortCircuitRenewCancel(TestClientRMTokens.java:353)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testShortCircuitRenewCancelWildcardAddress(TestClientRMTokens.java:294)
{noformat}

> Test failure for TestClientRMTokens
> ---
>
> Key: YARN-2634
> URL: https://issues.apache.org/jira/browse/YARN-2634
> Project: Hadoop YARN
>  Issue Type: Test
>Reporter: Junping Du
>Assignee: Jian He
>Priority: Blocker
>
> The test get failed as below:
> {noformat}
> ---
> Test set: 

[jira] [Updated] (YARN-5865) Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-5865:
--
Attachment: YARN-5865.branch-2.0002.patch

Updating branch-2 patch as there were some more unnecessary changes in earlier 
patch.Thanks [~rohithsharma]

> Retrospect updateApplicationPriority api to handle state store exception in 
> align with YARN-5611
> 
>
> Key: YARN-5865
> URL: https://issues.apache.org/jira/browse/YARN-5865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5865.0001.patch, YARN-5865.0002.patch, 
> YARN-5865.0003.patch, YARN-5865.0004.patch, YARN-5865.0005.patch, 
> YARN-5865.0006.patch, YARN-5865.branch-2.0001.patch, 
> YARN-5865.branch-2.0002.patch
>
>
> Post YARN-5611, revisit dynamic update of application priority logic with 
> respect to state store error handling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4206) Add life time value in Application report and CLI

[Rohith Sharma K S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-4206:

Attachment: YARN-4206.5.patch

Updated patch fixing Vinod review comments.


> Add life time value in Application report and CLI
> -
>
> Key: YARN-4206
> URL: https://issues.apache.org/jira/browse/YARN-4206
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: nijel
>Assignee: Rohith Sharma K S
> Attachments: YARN-4206.2.patch, YARN-4206.3.patch, YARN-4206.4.patch, 
> YARN-4206.5.patch, YARN-4506.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5918) Opportunistic scheduling allocate request failure when NM lost

[Bibin A Chundatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bibin A Chundatt updated YARN-5918:
---
Attachment: YARN-5918.0003.patch

Attaching patch handling comments

> Opportunistic scheduling allocate request failure when NM lost
> --
>
> Key: YARN-5918
> URL: https://issues.apache.org/jira/browse/YARN-5918
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-5918.0001.patch, YARN-5918.0002.patch, 
> YARN-5918.0003.patch
>
>
> Allocate request failure during Opportunistic container allocation when 
> nodemanager is lost 
> {noformat}
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root 
> OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS  
> APPID=application_1479637990302_0002
> CONTAINERID=container_e12_1479637990302_0002_01_06  
> RESOURCE=
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Removed node docker2:38297 clusterResource: 
> 2016-11-20 10:38:49,434 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 7 on 8030, call Call#35 Retry#0 
> org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 
> 172.17.0.2:51584
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNode(OpportunisticContainerAllocatorAMService.java:420)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNodes(OpportunisticContainerAllocatorAMService.java:412)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.getLeastLoadedNodes(OpportunisticContainerAllocatorAMService.java:402)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.allocate(OpportunisticContainerAllocatorAMService.java:236)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:846)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:789)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2539)
> 2016-11-20 10:38:50,824 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> container_e12_1479637990302_0002_01_02 Container Transitioned from 
> RUNNING to COMPLETED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5932) Retrospect moveToQueue in align with YARN-5611

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-5932:
--
Attachment: YARN-5932.v0.patch

Updating a version 0 patch to capture the flow.

Changes done:

- Currently this patch handles changes only for Capacity Scheduler
- A new pre-validate check is added for moveApplication in CapacityScheduler 
and Leaf/ParentQueue. This helps to ensure that the target queue can accept 
this application or not. Any failure could be handled at this stage itself.
- StateStore also updated with the queue name to which this app is moved to. We 
could update submission context for same.
- Finally perform real move operation. Ideally there should not be any 
exception thrown out. However with some corner cases, there is  very less 
chance to happen this. In such cases, we can revert from StateStore.

cc/[~jianhe][~rohithsharma]

> Retrospect moveToQueue in align with YARN-5611
> --
>
> Key: YARN-5932
> URL: https://issues.apache.org/jira/browse/YARN-5932
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler, resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5932.v0.patch
>
>
> All dynamic api's of an application's state change could follow a general 
> design approach. Currently priority and app timeouts are following this 
> approach all corner cases.
> *Steps*
> - Do a pre-validate check to ensure that changes are fine.
> - Update this information to state-store
> - Perform real move operation and update in-memory data structures.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5932) Retrospect moveToQueue in align with YARN-5611

[Sunil G (JIRA)]

Sunil G created YARN-5932:
-

 Summary: Retrospect moveToQueue in align with YARN-5611
 Key: YARN-5932
 URL: https://issues.apache.org/jira/browse/YARN-5932
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacity scheduler, resourcemanager
Reporter: Sunil G
Assignee: Sunil G


All dynamic api's of an application's state change could follow a general 
design approach. Currently priority and app timeouts are following this 
approach all corner cases.
*Steps*
- Do a pre-validate check to ensure that changes are fine.
- Update this information to state-store
- Perform real move operation and update in-memory data structures.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5918) Opportunistic scheduling allocate request failure when NM lost

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690057#comment-15690057
 ] 

Varun Saxena commented on YARN-5918:


Thanks [~bibinchundatt] for the latest patch.

We actually did not need such an elaborate test. We could have probably 
achieved the same by mocking the scheduler to simulate case of NPE. But its not 
necessarily bad to have some sort of E2E test case simulating the complete 
scenario.

Few comments.
# This test doesn't seem to belong to TestNodeQueueLoadMonitor. We can shift it 
to some other test class or create a new class if it cant fit in any.
# We are iterating 30 times over and sleeping for 100 ms in each run. This will 
unnecessarily make the test run for 3 seconds in loop. We can get rid of this 
loop and make run time of test faster. We can do it as under.
** You can set YarnConfiguration.NM_CONTAINER_QUEUING_SORTING_NODES_INTERVAL_MS 
in configuration as 100 ms instead of 1000 ms default.
** Move node added and node update events sent to AM Service outside the loop.
** We can check get OpportunisticContainerContext from FiCaSchedulerApp and use 
it to check against getNodeMap to have a deterministic test case and reduce 
test time.
** After invoking node add and update events, we can loop 10-20 times over(say) 
to send allocate with a sleep of say 50 ms. Break from the loop as soon as 
getNodeMap has 2 nodes. Now send remove node event to scheduler and then loop 
over to send allocate and wait till getNodeMap becomes 1. 
# Not related to your patch. In NodeQueueLoadMonitor we have some LOG.debug 
statements without isDebugEnabled guard. Maybe we can fix this here as well.

> Opportunistic scheduling allocate request failure when NM lost
> --
>
> Key: YARN-5918
> URL: https://issues.apache.org/jira/browse/YARN-5918
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-5918.0001.patch, YARN-5918.0002.patch
>
>
> Allocate request failure during Opportunistic container allocation when 
> nodemanager is lost 
> {noformat}
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root 
> OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS  
> APPID=application_1479637990302_0002
> CONTAINERID=container_e12_1479637990302_0002_01_06  
> RESOURCE=
> 2016-11-20 10:38:49,011 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Removed node docker2:38297 clusterResource: 
> 2016-11-20 10:38:49,434 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 7 on 8030, call Call#35 Retry#0 
> org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from 
> 172.17.0.2:51584
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNode(OpportunisticContainerAllocatorAMService.java:420)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNodes(OpportunisticContainerAllocatorAMService.java:412)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.getLeastLoadedNodes(OpportunisticContainerAllocatorAMService.java:402)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.allocate(OpportunisticContainerAllocatorAMService.java:236)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:846)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:789)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2539)
> 2016-11-20 10:38:50,824 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> container_e12_1479637990302_0002_01_02 Container Transitioned from 
> RUNNING to COMPLETED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-5818) Support the Docker Live Restore feature

[Shane Kumpf (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689892#comment-15689892
 ] 

Shane Kumpf commented on YARN-5818:
---

{{docker wait}} will have to be removed to support Docker live restore. 
Retrying the {{docker wait}} is brittle, as it requires parsing stderr and 
looking for a specific string which could change without notice. 

I propose we replace the {{docker wait}} approach with the following to support 
live restore:

# {{docker run}} to start the container.
# {{docker inspect}} to get the pid.
# Null signal ({{kill -0 pid}}) liveliness loop waiting for the container to 
complete.
# {{docker inspect}} the finished container for the exit code.
# Write the exitcode file to be picked up by the NM

The null signal loop has pitfalls, but this is the pattern we rely upon else 
where where wait/waitpid aren't possible (container re-acquisition on NM 
restart for example).

I'll put up a patch that does the above as a starting point. Please provide 
your thoughts on the approach.

> Support the Docker Live Restore feature
> ---
>
> Key: YARN-5818
> URL: https://issues.apache.org/jira/browse/YARN-5818
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Shane Kumpf
>
> Docker 1.12.x introduced the docker [Live 
> Restore|https://docs.docker.com/engine/admin/live-restore/] feature which 
> allows docker containers to survive docker daemon restarts/upgrades. Support 
> for this feature should be added to YARN to allow docker changes and upgrades 
> to be less impactful to existing containers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5865) Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689825#comment-15689825
 ] 

Hadoop QA commented on YARN-5865:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 
43s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 8 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
23s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 27s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 2 new + 643 unchanged - 2 fixed = 645 total (was 645) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m 
49s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 56s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | YARN-5865 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840227/YARN-5865.branch-2.0001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux e84e4564b3f5 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2 / 2d94f93 |
| Default Java | 1.7.0_121 |
| 

[jira] [Commented] (YARN-5926) clean up registry code for java 7/8

[Akira Ajisaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689775#comment-15689775
 ] 

Akira Ajisaka commented on YARN-5926:
-

Sure. I'm okay with changing the title.

> clean up registry code for java 7/8
> ---
>
> Key: YARN-5926
> URL: https://issues.apache.org/jira/browse/YARN-5926
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: YARN-5926-001.patch
>
>
> Clean up the registry code to stop the java 7/8 warnings



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4330) MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages.

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689399#comment-15689399
 ] 

Naganarasimha G R commented on YARN-4330:
-

test case failure are not related and findbugs do not run for 
*hadoop-yarn-server-tests* so went ahead with the commit, It applies 
successfully on trunk and branch-2, [~varun_saxena] can you check for 
branch-2.8 as its failing ?

> MiniYARNCluster is showing multiple  Failed to instantiate default resource 
> calculator warning messages.
> 
>
> Key: YARN-4330
> URL: https://issues.apache.org/jira/browse/YARN-4330
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test, yarn
>Affects Versions: 2.8.0
> Environment: OSX, JUnit
>Reporter: Steve Loughran
>Assignee: Varun Saxena
>Priority: Blocker
>  Labels: oct16-hard
> Attachments: YARN-4330.002.patch, YARN-4330.003.patch, 
> YARN-4330.004.patch, YARN-4330.01.patch
>
>
> Whenever I try to start a MiniYARNCluster on Branch-2 (commit #0b61cca), I 
> see multiple stack traces warning me that a resource calculator plugin could 
> not be created
> {code}
> (ResourceCalculatorPlugin.java:getResourceCalculatorPlugin(184)) - 
> java.lang.UnsupportedOperationException: Could not determine OS: Failed to 
> instantiate default resource calculator.
> java.lang.UnsupportedOperationException: Could not determine OS
> {code}
> This is a minicluster. It doesn't need resource calculation. It certainly 
> doesn't need test logs being cluttered with even more stack traces which will 
> only generate false alarms about tests failing. 
> There needs to be a way to turn this off, and the minicluster should have it 
> that way by default.
> Being ruthless and marking as a blocker, because its a fairly major 
> regression for anyone testing with the minicluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5911) DrainDispatcher does not drain all events on stop even if setDrainEventsOnStop is true

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689374#comment-15689374
 ] 

Varun Saxena commented on YARN-5911:


Thanks [~Naganarasimha] for the review and commit and thanks [~sandflee] and 
[~bibinchundatt] for reviews.

> DrainDispatcher does not drain all events on stop even if 
> setDrainEventsOnStop is true
> --
>
> Key: YARN-5911
> URL: https://issues.apache.org/jira/browse/YARN-5911
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5911.01.patch, YARN-5911.02.patch
>
>
> DrainDispatcher#serviceStop sets the stopped flag first before draining the 
> event queue.
> This means that the thread terminates as soon as it encounters stopped flag 
> as true and does not continue to process leftover events in queue, something 
> which it should do if setDrainEventsOnStop is set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org