[jira] [Commented] (YARN-2907) SchedulerNode#toString should print all resource detail instead of only memory.

2014-12-01 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229792#comment-14229792
 ] 

Hudson commented on YARN-2907:
--

FAILURE: Integrated in Hadoop-trunk-Commit #6616 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/6616/])
YARN-2907. SchedulerNode#toString should print all resource detail instead of 
only memory. (Contributed by Rohith) (junping_du: rev 
c732ed760e115192652280cd4b5799d49e62db7e)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
* hadoop-yarn-project/CHANGES.txt
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java


 SchedulerNode#toString should print all resource detail instead of only 
 memory.
 ---

 Key: YARN-2907
 URL: https://issues.apache.org/jira/browse/YARN-2907
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
Priority: Trivial
 Fix For: 2.7.0

 Attachments: 0001-YARN-2907.patch, YARN-2907.1.patch


 SchedulerNode#toString display only memory. It should display both memory and 
 cpu. It helps while analyzing logs
 {code}
   @Override
   public String toString() {
 return host:  + rmNode.getNodeAddress() +  #containers=
 + getNumContainers() +  available=
 + getAvailableResource().getMemory() +  used=
 + getUsedResource().getMemory();
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1156) Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values

2014-12-01 Thread Junping Du (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229824#comment-14229824
 ] 

Junping Du commented on YARN-1156:
--

Thanks [~ajisakaa] for reporting this bug and [~ozawa] for a quick fix for it. 
I think our adding allocatedMB and availableMB here is a good improvement for 
precisely information. However, we may want to keep allocatedGB and availableGB 
there for compatible with upstream projects (like Apache Ambari, etc.) or 3rd 
party tools. 
So I suggest we inc/dec MB (container size) to allocatedMB/availableMB, and 
calculate allocated/availableGB from allocatedMB/availableMB which make 
deviation here always in a small range ( 1G or 0.5G if round up).

 Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values
 -

 Key: YARN-1156
 URL: https://issues.apache.org/jira/browse/YARN-1156
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.1.0-beta
Reporter: Akira AJISAKA
Assignee: Tsuyoshi OZAWA
Priority: Minor
  Labels: metrics, newbie
 Fix For: 2.7.0

 Attachments: YARN-1156.1.patch, YARN-1156.2.patch


 AllocatedGB and AvailableGB metrics are now integer type. If there are four 
 times 500MB memory allocation to container, AllocatedGB is incremented four 
 times by {{(int)500/1024}}, which means 0. That is, the memory size allocated 
 is actually 2000MB, but the metrics shows 0GB. Let's use float type for these 
 metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2907) SchedulerNode#toString should print all resource detail instead of only memory.

2014-12-01 Thread Rohith (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229906#comment-14229906
 ] 

Rohith commented on YARN-2907:
--

Thanks [~djp] for review and committing patch:-)

 SchedulerNode#toString should print all resource detail instead of only 
 memory.
 ---

 Key: YARN-2907
 URL: https://issues.apache.org/jira/browse/YARN-2907
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
Priority: Trivial
 Fix For: 2.7.0

 Attachments: 0001-YARN-2907.patch, YARN-2907.1.patch


 SchedulerNode#toString display only memory. It should display both memory and 
 cpu. It helps while analyzing logs
 {code}
   @Override
   public String toString() {
 return host:  + rmNode.getNodeAddress() +  #containers=
 + getNumContainers() +  available=
 + getAvailableResource().getMemory() +  used=
 + getUsedResource().getMemory();
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2189) Admin service for cache manager

2014-12-01 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230127#comment-14230127
 ] 

Chris Trezzo commented on YARN-2189:


The test failure was due to an unrelated ConnectionException. The patch should 
be good as is.

 Admin service for cache manager
 ---

 Key: YARN-2189
 URL: https://issues.apache.org/jira/browse/YARN-2189
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Chris Trezzo
Assignee: Chris Trezzo
 Attachments: YARN-2189-trunk-v1.patch, YARN-2189-trunk-v2.patch, 
 YARN-2189-trunk-v3.patch, YARN-2189-trunk-v4.patch, YARN-2189-trunk-v5.patch


 Implement the admin service for the shared cache manager. This service is 
 responsible for handling administrative commands such as manually running a 
 cleaner task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2136) RMStateStore can explicitly handle store/update events when fenced

2014-12-01 Thread Jian He (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230215#comment-14230215
 ] 

Jian He commented on YARN-2136:
---

[~varun_saxena], thanks for updating!  looks good overall,
one minor comment: we may break out of the while loop if it's at FENCED state.
{code}
  if(!isFencedState()) { 
doMultiWithRetries(emptyOpList);
  }
{code}

 RMStateStore can explicitly handle store/update events when fenced
 --

 Key: YARN-2136
 URL: https://issues.apache.org/jira/browse/YARN-2136
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Jian He
Assignee: Varun Saxena
 Fix For: 2.7.0

 Attachments: YARN-2136.002.patch, YARN-2136.003.patch, 
 YARN-2136.004.patch, YARN-2136.patch


 RMStateStore can choose to handle/ignore store/update events upfront instead 
 of invoking more ZK operations if state store is at fenced state. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2674) Distributed shell AM may re-launch containers if RM work preserving restart happens

2014-12-01 Thread Jian He (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230275#comment-14230275
 ] 

Jian He commented on YARN-2674:
---

[~chenchun], thanks for your patch. The patch should work fine.
Maybe a more standard way is to use the {{AMRMClient#getMatchingRequests}} to 
find out the matching resourceRequest and then call the removeContainerRequest 
method ?

 Distributed shell AM may re-launch containers if RM work preserving restart 
 happens
 ---

 Key: YARN-2674
 URL: https://issues.apache.org/jira/browse/YARN-2674
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Chun Chen
 Attachments: YARN-2674.1.patch


 Currently, if RM work preserving restart happens while distributed shell is 
 running, distribute shell AM may re-launch all the containers, including 
 new/running/complete. We must make sure it won't re-launch the 
 running/complete containers.
 We need to remove allocated containers from 
 AMRMClientImpl#remoteRequestsTable once AM receive them from RM. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2880) Add a test in TestRMRestart to make sure node labels will be recovered if it is enabled

2014-12-01 Thread Jian He (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230283#comment-14230283
 ] 

Jian He commented on YARN-2880:
---

looks good, re-kick jenkins

 Add a test in TestRMRestart to make sure node labels will be recovered if it 
 is enabled
 ---

 Key: YARN-2880
 URL: https://issues.apache.org/jira/browse/YARN-2880
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Wangda Tan
Assignee: Rohith
 Attachments: 0001-YARN-2880.patch, YARN-2880.1.patch


 As suggested by [~ozawa], 
 [link|https://issues.apache.org/jira/browse/YARN-2800?focusedCommentId=14217569page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14217569].
  We should have a such test to make sure there will be no regression



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (YARN-2913) Fair scheduler should have ability to set MaxResourceDefault for each queue

2014-12-01 Thread Siqi Li (JIRA) 
Siqi Li created YARN-2913:
-

 Summary: Fair scheduler should have ability to set 
MaxResourceDefault for each queue
 Key: YARN-2913
 URL: https://issues.apache.org/jira/browse/YARN-2913
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Siqi Li






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2913) Fair scheduler should have ability to set MaxResourceDefault for each queue

2014-12-01 Thread Siqi Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siqi Li updated YARN-2913:
--
Description: Queues that are created on the fly have the max resource of 
the entire cluster. Fair Scheduler should have a default maxResource to control 
the maxResource of those queues

 Fair scheduler should have ability to set MaxResourceDefault for each queue
 ---

 Key: YARN-2913
 URL: https://issues.apache.org/jira/browse/YARN-2913
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Siqi Li

 Queues that are created on the fly have the max resource of the entire 
 cluster. Fair Scheduler should have a default maxResource to control the 
 maxResource of those queues



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2894) When ACL's are enabled, if RM switches then application can not be viewed from web.

2014-12-01 Thread Jian He (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230341#comment-14230341
 ] 

Jian He commented on YARN-2894:
---

make sense. patch looks good.
Could you also check if binding to RMContext has similar problem ? It seems 
fine to me. Or we can also getRMContext from the rm object for future-proof.
{{bind(RMContext.class).toInstance(rm.getRMContext());}}

 When ACL's are enabled, if RM switches then application can not be viewed 
 from web.
 ---

 Key: YARN-2894
 URL: https://issues.apache.org/jira/browse/YARN-2894
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
 Fix For: 2.7.0

 Attachments: YARN-2894.patch


 Binding aclManager to RMWebApp would cause problem if RM is switched. There 
 could be some validation check may fail.
 I think , we should not bind aclManager for RMWebApp, instead we should get 
 from RM instance.
 In RMWebApp,
 {code}
 if (rm != null) {
   bind(ResourceManager.class).toInstance(rm);
   bind(RMContext.class).toInstance(rm.getRMContext());
   bind(ApplicationACLsManager.class).toInstance(
   rm.getApplicationACLsManager());
   bind(QueueACLsManager.class).toInstance(rm.getQueueACLsManager());
 }
 {code}
 and in AppBlock#render below check may fail(Need to test and confirm)
 {code}
if (callerUGI != null
  !(this.aclsManager.checkAccess(callerUGI,
 ApplicationAccessType.VIEW_APP, app.getUser(), appID) ||
  this.queueACLsManager.checkAccess(callerUGI,
 QueueACL.ADMINISTER_QUEUE, app.getQueue( {
   puts(You (User  + remoteUser
   + ) are not authorized to view application  + appID);
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2894) When ACL's are enabled, if RM switches then application can not be viewed from web.

2014-12-01 Thread Jian He (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230345#comment-14230345
 ] 

Jian He commented on YARN-2894:
---

Minor comment: unused imports in RMWebApp.java

 When ACL's are enabled, if RM switches then application can not be viewed 
 from web.
 ---

 Key: YARN-2894
 URL: https://issues.apache.org/jira/browse/YARN-2894
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
 Fix For: 2.7.0

 Attachments: YARN-2894.patch


 Binding aclManager to RMWebApp would cause problem if RM is switched. There 
 could be some validation check may fail.
 I think , we should not bind aclManager for RMWebApp, instead we should get 
 from RM instance.
 In RMWebApp,
 {code}
 if (rm != null) {
   bind(ResourceManager.class).toInstance(rm);
   bind(RMContext.class).toInstance(rm.getRMContext());
   bind(ApplicationACLsManager.class).toInstance(
   rm.getApplicationACLsManager());
   bind(QueueACLsManager.class).toInstance(rm.getQueueACLsManager());
 }
 {code}
 and in AppBlock#render below check may fail(Need to test and confirm)
 {code}
if (callerUGI != null
  !(this.aclsManager.checkAccess(callerUGI,
 ApplicationAccessType.VIEW_APP, app.getUser(), appID) ||
  this.queueACLsManager.checkAccess(callerUGI,
 QueueACL.ADMINISTER_QUEUE, app.getQueue( {
   puts(You (User  + remoteUser
   + ) are not authorized to view application  + appID);
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230364#comment-14230364
 ] 

Hadoop QA commented on YARN-2890:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12684437/YARN-2890.patch
  against trunk revision c732ed7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:

  org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
  
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5959//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5959//console

This message is automatically generated.

 MiniMRYarnCluster should turn on timeline service if configured to do so
 

 Key: YARN-2890
 URL: https://issues.apache.org/jira/browse/YARN-2890
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mit Desai
Assignee: Mit Desai
 Fix For: 2.6.1

 Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch


 Currently the MiniMRYarnCluster does not consider the configuration value for 
 enabling timeline service before starting. The MiniYarnCluster should only 
 start the timeline service if it is configured to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so

2014-12-01 Thread Zhijie Shen (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230378#comment-14230378
 ] 

Zhijie Shen commented on YARN-2890:
---

In the scenario of
{code}
conf.setBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED, false);
conf.setBoolean(MRJobConfig.MAPREDUCE_JOB_EMIT_TIMELINE_DATA, true);
{code}

Would you please run through an MR job to validate the job can run successfully?

 MiniMRYarnCluster should turn on timeline service if configured to do so
 

 Key: YARN-2890
 URL: https://issues.apache.org/jira/browse/YARN-2890
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mit Desai
Assignee: Mit Desai
 Fix For: 2.6.1

 Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch


 Currently the MiniMRYarnCluster does not consider the configuration value for 
 enabling timeline service before starting. The MiniYarnCluster should only 
 start the timeline service if it is configured to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so

2014-12-01 Thread Zhijie Shen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated YARN-2890:
--
Fix Version/s: (was: 2.6.1)
   2.7.0

 MiniMRYarnCluster should turn on timeline service if configured to do so
 

 Key: YARN-2890
 URL: https://issues.apache.org/jira/browse/YARN-2890
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mit Desai
Assignee: Mit Desai
 Fix For: 2.7.0

 Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch


 Currently the MiniMRYarnCluster does not consider the configuration value for 
 enabling timeline service before starting. The MiniYarnCluster should only 
 start the timeline service if it is configured to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so

2014-12-01 Thread Mit Desai (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230420#comment-14230420
 ] 

Mit Desai commented on YARN-2890:
-

{quote}
-1 core tests. The patch failed these unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:

org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
{quote}


 MiniMRYarnCluster should turn on timeline service if configured to do so
 

 Key: YARN-2890
 URL: https://issues.apache.org/jira/browse/YARN-2890
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mit Desai
Assignee: Mit Desai
 Fix For: 2.7.0

 Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch


 Currently the MiniMRYarnCluster does not consider the configuration value for 
 enabling timeline service before starting. The MiniYarnCluster should only 
 start the timeline service if it is configured to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so

2014-12-01 Thread Mit Desai (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230422#comment-14230422
 ] 

Mit Desai commented on YARN-2890:
-

Tried running these tests without the patch. They still fail. Does not look 
like a side effect of this patch.

bq. Would you please run through an MR job to validate the job can run 
successfully?

Nice catch. I thought this scenario was already covered. Will update the patch.

 MiniMRYarnCluster should turn on timeline service if configured to do so
 

 Key: YARN-2890
 URL: https://issues.apache.org/jira/browse/YARN-2890
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mit Desai
Assignee: Mit Desai
 Fix For: 2.7.0

 Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch


 Currently the MiniMRYarnCluster does not consider the configuration value for 
 enabling timeline service before starting. The MiniYarnCluster should only 
 start the timeline service if it is configured to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2880) Add a test in TestRMRestart to make sure node labels will be recovered if it is enabled

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230423#comment-14230423
 ] 

Hadoop QA commented on YARN-2880:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12683864/YARN-2880.1.patch
  against trunk revision c732ed7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

  org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
  
org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5960//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5960//console

This message is automatically generated.

 Add a test in TestRMRestart to make sure node labels will be recovered if it 
 is enabled
 ---

 Key: YARN-2880
 URL: https://issues.apache.org/jira/browse/YARN-2880
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Wangda Tan
Assignee: Rohith
 Attachments: 0001-YARN-2880.patch, YARN-2880.1.patch


 As suggested by [~ozawa], 
 [link|https://issues.apache.org/jira/browse/YARN-2800?focusedCommentId=14217569page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14217569].
  We should have a such test to make sure there will be no regression



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2894) When ACL's are enabled, if RM switches then application can not be viewed from web.

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230425#comment-14230425
 ] 

Hadoop QA commented on YARN-2894:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12683865/YARN-2894.patch
  against trunk revision c732ed7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

  
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
  
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
  
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
  
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
  
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
  
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
  
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
  
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5961//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5961//console

This message is automatically generated.

 When ACL's are enabled, if RM switches then application can not be viewed 
 from web.
 ---

 Key: YARN-2894
 URL: https://issues.apache.org/jira/browse/YARN-2894
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
 Fix For: 2.7.0

 Attachments: YARN-2894.patch


 Binding aclManager to RMWebApp would cause problem if RM is switched. There 
 could be some validation check may fail.
 I think , we should not bind aclManager for RMWebApp, instead we should get 
 from RM instance.
 In RMWebApp,
 {code}
 if (rm != null) {
   bind(ResourceManager.class).toInstance(rm);
   bind(RMContext.class).toInstance(rm.getRMContext());
   bind(ApplicationACLsManager.class).toInstance(
   rm.getApplicationACLsManager());
   bind(QueueACLsManager.class).toInstance(rm.getQueueACLsManager());
 }
 {code}
 and in AppBlock#render below check may fail(Need to test and confirm)
 {code}
if (callerUGI != null
  !(this.aclsManager.checkAccess(callerUGI,
 ApplicationAccessType.VIEW_APP, app.getUser(), appID) ||
  this.queueACLsManager.checkAccess(callerUGI,
 QueueACL.ADMINISTER_QUEUE, app.getQueue( {
   puts(You (User  + remoteUser
   + ) are not authorized to view application  + appID);
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (YARN-2914) Potential race condition in ClientSCMMetrics#getInstance()

2014-12-01 Thread Ted Yu (JIRA) 
Ted Yu created YARN-2914:


 Summary: Potential race condition in ClientSCMMetrics#getInstance()
 Key: YARN-2914
 URL: https://issues.apache.org/jira/browse/YARN-2914
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  public static ClientSCMMetrics getInstance() {
ClientSCMMetrics topMetrics = Singleton.INSTANCE.impl;
if (topMetrics == null) {
  throw new IllegalStateException(
{code}
getInstance() doesn't hold lock on Singleton.this
This may result in IllegalStateException being thrown prematurely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so

2014-12-01 Thread Mit Desai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mit Desai updated YARN-2890:

Attachment: YARN-2890.patch

Attaching the patch that includes a MR job running with timeline service 
disabled. Also cleaned up couple debug statements.

 MiniMRYarnCluster should turn on timeline service if configured to do so
 

 Key: YARN-2890
 URL: https://issues.apache.org/jira/browse/YARN-2890
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mit Desai
Assignee: Mit Desai
 Fix For: 2.7.0

 Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, 
 YARN-2890.patch


 Currently the MiniMRYarnCluster does not consider the configuration value for 
 enabling timeline service before starting. The MiniYarnCluster should only 
 start the timeline service if it is configured to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (YARN-2914) Potential race condition in ClientSCMMetrics#getInstance()

2014-12-01 Thread Varun Saxena (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena reassigned YARN-2914:
--

Assignee: Varun Saxena

 Potential race condition in ClientSCMMetrics#getInstance()
 --

 Key: YARN-2914
 URL: https://issues.apache.org/jira/browse/YARN-2914
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Varun Saxena
Priority: Minor

 {code}
   public static ClientSCMMetrics getInstance() {
 ClientSCMMetrics topMetrics = Singleton.INSTANCE.impl;
 if (topMetrics == null) {
   throw new IllegalStateException(
 {code}
 getInstance() doesn't hold lock on Singleton.this
 This may result in IllegalStateException being thrown prematurely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2765) Add leveldb-based implementation for RMStateStore

2014-12-01 Thread Jason Lowe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated YARN-2765:
-
Attachment: YARN-2765v3.patch

Thanks for the review, Jian!

bq. Patch needs updated on top of YARN-2404.
Done.

{quote}
DB_SCHEMA_VERSION_KEY - VERSION_NODE
EPOCH_KEY - EPOCH_NODE
{quote}
Done.

bq. RM_DT_SEQUENCE_NUMBER_KEY: use DELEGATION_TOKEN_SEQUENCE_NUMBER_PREFIX for 
“RMDTSequentialNumber”

I did not make this change.  Unlike the filesystem state store we are not using 
the key to encode the data but rather just as a static key that is atomically 
updated with new values.  Therefore it's not appropriate to use a key that ends 
in a prefix, IMHO.  This store is closer to what the ZK store is doing which 
also uses a separate key string for the sequence number.

bq. this check is always true and so unnecessary, as attemptState is earlier 
initialized with attemptId

Left this check since after the refactoring for YARN-2404 the check will verify 
the attempt ID parsed from the key matches the attempt ID parsed from the value.

{quote}
Some naming suggestions:
loadRMAppAttemptState- createAttemptState
loadRMAppState(String appIdStr, byte[] data) - createApplicationState
String itemName = key.substring(keyPrefix.length()); rename itemName to 
attemptId, and keyPrefix to attemptNodePrefix
{quote}
Done.

bq. Question: does the levelDB client API handle retry ?
No, there are no retries in leveldb itself AFAIK.  The retries are handled by 
the OS/driver/device layers, as we are hitting the OS filesystem rather than 
some network layer like HDFS or ZK does.  Retries would be highly unlikely to 
be successful at the leveldb or app level.

 Add leveldb-based implementation for RMStateStore
 -

 Key: YARN-2765
 URL: https://issues.apache.org/jira/browse/YARN-2765
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Jason Lowe
Assignee: Jason Lowe
 Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch


 It would be nice to have a leveldb option to the resourcemanager recovery 
 store. Leveldb would provide some benefits over the existing filesystem store 
 such as better support for atomic operations, fewer I/O ops per state update, 
 and far fewer total files on the filesystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2891) Failed Container Executor does not provide a clear error message

2014-12-01 Thread Dustin Cote (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dustin Cote updated YARN-2891:
--
Attachment: YARN-2891-1.patch

Attaching a patch file because I'm not really sure how to submit it otherwise 
for review.

 Failed Container Executor does not provide a clear error message
 

 Key: YARN-2891
 URL: https://issues.apache.org/jira/browse/YARN-2891
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.5.1
 Environment: any
Reporter: Dustin Cote
Assignee: Dustin Cote
Priority: Minor
 Attachments: YARN-2891-1.patch


 When checking access to directories, the container executor does not provide 
 clear information on which directory actually could not be accessed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-1621) Add CLI to list rows of task attempt ID, container ID, host of container, state of container

2014-12-01 Thread JIRA 

 [ 
https://issues.apache.org/jira/browse/YARN-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bartosz Ługowski updated YARN-1621:
---
Attachment: YARN-1621.2.patch

Patch update.

 Add CLI to list rows of task attempt ID, container ID, host of container, 
 state of container
 --

 Key: YARN-1621
 URL: https://issues.apache.org/jira/browse/YARN-1621
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.2.0
Reporter: Tassapol Athiapinya
 Fix For: 2.7.0

 Attachments: YARN-1621.1.patch, YARN-1621.2.patch


 As more applications are moved to YARN, we need generic CLI to list rows of 
 task attempt ID, container ID, host of container, state of container. Today 
 if YARN application running in a container does hang, there is no way to find 
 out more info because a user does not know where each attempt is running in.
 For each running application, it is useful to differentiate between 
 running/succeeded/failed/killed containers.
  
 {code:title=proposed yarn cli}
 $ yarn application -list-containers -applicationId appId [-containerState 
 state of container]
 where containerState is optional filter to list container in given state only.
 container state can be running/succeeded/killed/failed/all.
 A user can specify more than one container state at once e.g. KILLED,FAILED.
 task attempt ID container ID host of container state of container 
 {code}
 CLI should work with running application/completed application. If a 
 container runs many task attempts, all attempts should be shown. That will 
 likely be the case of Tez container-reuse application.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2891) Failed Container Executor does not provide a clear error message

2014-12-01 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated YARN-2891:
--
Hadoop Flags: Reviewed

 Failed Container Executor does not provide a clear error message
 

 Key: YARN-2891
 URL: https://issues.apache.org/jira/browse/YARN-2891
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.5.1
 Environment: any
Reporter: Dustin Cote
Assignee: Dustin Cote
Priority: Minor
 Attachments: YARN-2891-1.patch


 When checking access to directories, the container executor does not provide 
 clear information on which directory actually could not be accessed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)

2014-12-01 Thread Mit Desai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mit Desai updated YARN-2900:

Attachment: YARN-2900.patch

[~zjshen] I am still working on the unit test. Meanwhile would you like to take 
a look at the changes that you suggested?

 Application (Attempt and Container) Not Found in AHS results in Internal 
 Server Error (500)
 ---

 Key: YARN-2900
 URL: https://issues.apache.org/jira/browse/YARN-2900
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Jonathan Eagles
Assignee: Mit Desai
 Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch


 Caused by: java.lang.NullPointerException
   at 
 org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128)
   at 
 org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118)
   at 
 org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222)
   at 
 org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679)
   at 
 org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218)
   ... 59 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2905) AggregatedLogsBlock page can infinitely loop if the aggregated log file is corrupted

2014-12-01 Thread Jason Lowe (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230571#comment-14230571
 ] 

Jason Lowe commented on YARN-2905:
--

+1 lgtm.  Committing this.

 AggregatedLogsBlock page can infinitely loop if the aggregated log file is 
 corrupted
 

 Key: YARN-2905
 URL: https://issues.apache.org/jira/browse/YARN-2905
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Jason Lowe
Assignee: Varun Saxena
Priority: Blocker
 Attachments: YARN-2905.patch


 If the AggregatedLogsBlock page tries to serve up a portion of a log file 
 that has been corrupted (e.g.: like the case that was fixed by YARN-2724) 
 then it can spin forever trying to seek to the targeted log segment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2905) AggregatedLogsBlock page can infinitely loop if the aggregated log file is corrupted

2014-12-01 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230575#comment-14230575
 ] 

Hudson commented on YARN-2905:
--

FAILURE: Integrated in Hadoop-trunk-Commit #6619 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/6619/])
YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated log 
file is corrupted. Contributed by Varun Saxena (jlowe: rev 
0f9528b99addbb0fd9a19d84db22a8c8e934b05f)
* hadoop-yarn-project/CHANGES.txt
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/log/AggregatedLogsBlock.java


 AggregatedLogsBlock page can infinitely loop if the aggregated log file is 
 corrupted
 

 Key: YARN-2905
 URL: https://issues.apache.org/jira/browse/YARN-2905
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Jason Lowe
Assignee: Varun Saxena
Priority: Blocker
 Attachments: YARN-2905.patch


 If the AggregatedLogsBlock page tries to serve up a portion of a log file 
 that has been corrupted (e.g.: like the case that was fixed by YARN-2724) 
 then it can spin forever trying to seek to the targeted log segment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2891) Failed Container Executor does not provide a clear error message

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230584#comment-14230584
 ] 

Hadoop QA commented on YARN-2891:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12684477/YARN-2891-1.patch
  against trunk revision 2b30fb1.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5965//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5965//console

This message is automatically generated.

 Failed Container Executor does not provide a clear error message
 

 Key: YARN-2891
 URL: https://issues.apache.org/jira/browse/YARN-2891
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.5.1
 Environment: any
Reporter: Dustin Cote
Assignee: Dustin Cote
Priority: Minor
 Attachments: YARN-2891-1.patch


 When checking access to directories, the container executor does not provide 
 clear information on which directory actually could not be accessed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1621) Add CLI to list rows of task attempt ID, container ID, host of container, state of container

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230587#comment-14230587
 ] 

Hadoop QA commented on YARN-1621:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12684481/YARN-1621.2.patch
  against trunk revision 2b30fb1.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:

  org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
  
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5964//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5964//console

This message is automatically generated.

 Add CLI to list rows of task attempt ID, container ID, host of container, 
 state of container
 --

 Key: YARN-1621
 URL: https://issues.apache.org/jira/browse/YARN-1621
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.2.0
Reporter: Tassapol Athiapinya
 Fix For: 2.7.0

 Attachments: YARN-1621.1.patch, YARN-1621.2.patch


 As more applications are moved to YARN, we need generic CLI to list rows of 
 task attempt ID, container ID, host of container, state of container. Today 
 if YARN application running in a container does hang, there is no way to find 
 out more info because a user does not know where each attempt is running in.
 For each running application, it is useful to differentiate between 
 running/succeeded/failed/killed containers.
  
 {code:title=proposed yarn cli}
 $ yarn application -list-containers -applicationId appId [-containerState 
 state of container]
 where containerState is optional filter to list container in given state only.
 container state can be running/succeeded/killed/failed/all.
 A user can specify more than one container state at once e.g. KILLED,FAILED.
 task attempt ID container ID host of container state of container 
 {code}
 CLI should work with running application/completed application. If a 
 container runs many task attempts, all attempts should be shown. That will 
 likely be the case of Tez container-reuse application.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230598#comment-14230598
 ] 

Hadoop QA commented on YARN-2765:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12684476/YARN-2765v3.patch
  against trunk revision 86e3993.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5963//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5963//console

This message is automatically generated.

 Add leveldb-based implementation for RMStateStore
 -

 Key: YARN-2765
 URL: https://issues.apache.org/jira/browse/YARN-2765
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Jason Lowe
Assignee: Jason Lowe
 Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch


 It would be nice to have a leveldb option to the resourcemanager recovery 
 store. Leveldb would provide some benefits over the existing filesystem store 
 such as better support for atomic operations, fewer I/O ops per state update, 
 and far fewer total files on the filesystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230632#comment-14230632
 ] 

Hadoop QA commented on YARN-2890:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12684475/YARN-2890.patch
  against trunk revision 86e3993.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:

  org.apache.hadoop.mapred.TestMiniMRChildTask
  org.apache.hadoop.mapred.TestJobSysDirWithDFS
  org.apache.hadoop.mapred.TestMiniMRBringup
  
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels

  The following test timeouts occurred in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:

org.apache.hadoop.mapred.lib.aggregate.TestAggregates

  The test build failed in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5962//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5962//console

This message is automatically generated.

 MiniMRYarnCluster should turn on timeline service if configured to do so
 

 Key: YARN-2890
 URL: https://issues.apache.org/jira/browse/YARN-2890
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mit Desai
Assignee: Mit Desai
 Fix For: 2.7.0

 Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, 
 YARN-2890.patch


 Currently the MiniMRYarnCluster does not consider the configuration value for 
 enabling timeline service before starting. The MiniYarnCluster should only 
 start the timeline service if it is configured to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1156) Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values

2014-12-01 Thread Tsuyoshi OZAWA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230635#comment-14230635
 ] 

Tsuyoshi OZAWA commented on YARN-1156:
--

Make sense. Updating.

 Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values
 -

 Key: YARN-1156
 URL: https://issues.apache.org/jira/browse/YARN-1156
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.1.0-beta
Reporter: Akira AJISAKA
Assignee: Tsuyoshi OZAWA
Priority: Minor
  Labels: metrics, newbie
 Fix For: 2.7.0

 Attachments: YARN-1156.1.patch, YARN-1156.2.patch


 AllocatedGB and AvailableGB metrics are now integer type. If there are four 
 times 500MB memory allocation to container, AllocatedGB is incremented four 
 times by {{(int)500/1024}}, which means 0. That is, the memory size allocated 
 is actually 2000MB, but the metrics shows 0GB. Let's use float type for these 
 metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so

2014-12-01 Thread Mit Desai (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230691#comment-14230691
 ] 

Mit Desai commented on YARN-2890:
-

Verified that the failures are not caused by the patch.

 MiniMRYarnCluster should turn on timeline service if configured to do so
 

 Key: YARN-2890
 URL: https://issues.apache.org/jira/browse/YARN-2890
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mit Desai
Assignee: Mit Desai
 Fix For: 2.7.0

 Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, 
 YARN-2890.patch


 Currently the MiniMRYarnCluster does not consider the configuration value for 
 enabling timeline service before starting. The MiniYarnCluster should only 
 start the timeline service if it is configured to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore

2014-12-01 Thread Jian He (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230754#comment-14230754
 ] 

Jian He commented on YARN-2765:
---

looks good, +1

 Add leveldb-based implementation for RMStateStore
 -

 Key: YARN-2765
 URL: https://issues.apache.org/jira/browse/YARN-2765
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Jason Lowe
Assignee: Jason Lowe
 Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch


 It would be nice to have a leveldb option to the resourcemanager recovery 
 store. Leveldb would provide some benefits over the existing filesystem store 
 such as better support for atomic operations, fewer I/O ops per state update, 
 and far fewer total files on the filesystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2032) Implement a scalable, available TimelineStore using HBase

2014-12-01 Thread Li Lu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Li Lu updated YARN-2032:

Target Version/s: 2.7.0  (was: 2.6.0)

 Implement a scalable, available TimelineStore using HBase
 -

 Key: YARN-2032
 URL: https://issues.apache.org/jira/browse/YARN-2032
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Li Lu
 Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch, 
 YARN-2032-branch2-2.patch


 As discussed on YARN-1530, we should pursue implementing a scalable, 
 available Timeline store using HBase.
 One goal is to reuse most of the code from the levelDB Based store - 
 YARN-1635.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore

2014-12-01 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230782#comment-14230782
 ] 

Hudson commented on YARN-2765:
--

FAILURE: Integrated in Hadoop-trunk-Commit #6622 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/6622/])
YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by 
Jason Lowe (jianhe: rev a7fba0bc28764e0fb36c335ea60cc58079fe007f)
* hadoop-yarn-project/CHANGES.txt
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestLeveldbRMStateStore.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java


 Add leveldb-based implementation for RMStateStore
 -

 Key: YARN-2765
 URL: https://issues.apache.org/jira/browse/YARN-2765
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Jason Lowe
Assignee: Jason Lowe
 Fix For: 2.7.0

 Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch


 It would be nice to have a leveldb option to the resourcemanager recovery 
 store. Leveldb would provide some benefits over the existing filesystem store 
 such as better support for atomic operations, fewer I/O ops per state update, 
 and far fewer total files on the filesystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2880) Add a test in TestRMRestart to make sure node labels will be recovered if it is enabled

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230838#comment-14230838
 ] 

Hadoop QA commented on YARN-2880:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12683864/YARN-2880.1.patch
  against trunk revision a7fba0b.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

  org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5966//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5966//console

This message is automatically generated.

 Add a test in TestRMRestart to make sure node labels will be recovered if it 
 is enabled
 ---

 Key: YARN-2880
 URL: https://issues.apache.org/jira/browse/YARN-2880
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Wangda Tan
Assignee: Rohith
 Attachments: 0001-YARN-2880.patch, YARN-2880.1.patch


 As suggested by [~ozawa], 
 [link|https://issues.apache.org/jira/browse/YARN-2800?focusedCommentId=14217569page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14217569].
  We should have a such test to make sure there will be no regression



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-1156) Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values

2014-12-01 Thread Tsuyoshi OZAWA (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated YARN-1156:
-
Attachment: YARN-1156.3.patch

Updated a patch to address Junping's commnt.

 Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values
 -

 Key: YARN-1156
 URL: https://issues.apache.org/jira/browse/YARN-1156
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.1.0-beta
Reporter: Akira AJISAKA
Assignee: Tsuyoshi OZAWA
Priority: Minor
  Labels: metrics, newbie
 Fix For: 2.7.0

 Attachments: YARN-1156.1.patch, YARN-1156.2.patch, YARN-1156.3.patch


 AllocatedGB and AvailableGB metrics are now integer type. If there are four 
 times 500MB memory allocation to container, AllocatedGB is incremented four 
 times by {{(int)500/1024}}, which means 0. That is, the memory size allocated 
 is actually 2000MB, but the metrics shows 0GB. Let's use float type for these 
 metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2880) Add a test in TestRMRestart to make sure node labels will be recovered if it is enabled

2014-12-01 Thread Rohith (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230861#comment-14230861
 ] 

Rohith commented on YARN-2880:
--

I am looking into test failures

 Add a test in TestRMRestart to make sure node labels will be recovered if it 
 is enabled
 ---

 Key: YARN-2880
 URL: https://issues.apache.org/jira/browse/YARN-2880
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Wangda Tan
Assignee: Rohith
 Attachments: 0001-YARN-2880.patch, YARN-2880.1.patch


 As suggested by [~ozawa], 
 [link|https://issues.apache.org/jira/browse/YARN-2800?focusedCommentId=14217569page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14217569].
  We should have a such test to make sure there will be no regression



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2837) Timeline server needs to recover the timeline DT when restarting

2014-12-01 Thread Zhijie Shen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated YARN-2837:
--
Attachment: YARN-2837.2.patch

Created a new patch to do the following improvement:

1. Add leveldb version check and a related test case.

2. Catch DBException.

3. Refactor the state store class names.

4.Check the newly added numeric config.

5. Move the common code out of LeveldbTimelineStore and put it into the util 
lib.

 Timeline server needs to recover the timeline DT when restarting
 

 Key: YARN-2837
 URL: https://issues.apache.org/jira/browse/YARN-2837
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
 Attachments: YARN-2837.1.patch, YARN-2837.2.patch


 Timeline server needs to recover the stateful information when restarting as 
 RM/NM/JHS does now. So far the stateful information only includes the 
 timeline DT. Without recovery, the timeline DT of the existing YARN apps is 
 not long valid, and cannot be renewed any more after the timeline server is 
 restarted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2762) RMAdminCLI node-labels-related args should be trimmed and checked before sending to RM

2014-12-01 Thread Rohith (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230864#comment-14230864
 ] 

Rohith commented on YARN-2762:
--

Hi [~leftnoteasy] Kindly review the patch.

 RMAdminCLI node-labels-related args should be trimmed and checked before 
 sending to RM
 --

 Key: YARN-2762
 URL: https://issues.apache.org/jira/browse/YARN-2762
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Rohith
Assignee: Rohith
Priority: Minor
 Attachments: YARN-2762.1.patch, YARN-2762.2.patch, YARN-2762.patch


 All NodeLabel args validation's are done at server side. The same can be done 
 at RMAdminCLI so that unnecessary RPC calls can be avoided.
 And for the input such as x,y,,z,, no need to add empty string instead can 
 be skipped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2894) When ACL's are enabled, if RM switches then application can not be viewed from web.

2014-12-01 Thread Rohith (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230880#comment-14230880
 ] 

Rohith commented on YARN-2894:
--

bq. Could you also check if binding to RMContext has similar problem ? It seems 
fine to me
This particular scenario I verified. This will not cuase any problem since 
RMContext created per JVM and activeservice instaces are updated.

bq. we can also getRMContext from the rm object for future-proof
Agree.


 When ACL's are enabled, if RM switches then application can not be viewed 
 from web.
 ---

 Key: YARN-2894
 URL: https://issues.apache.org/jira/browse/YARN-2894
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
 Fix For: 2.7.0

 Attachments: YARN-2894.patch


 Binding aclManager to RMWebApp would cause problem if RM is switched. There 
 could be some validation check may fail.
 I think , we should not bind aclManager for RMWebApp, instead we should get 
 from RM instance.
 In RMWebApp,
 {code}
 if (rm != null) {
   bind(ResourceManager.class).toInstance(rm);
   bind(RMContext.class).toInstance(rm.getRMContext());
   bind(ApplicationACLsManager.class).toInstance(
   rm.getApplicationACLsManager());
   bind(QueueACLsManager.class).toInstance(rm.getQueueACLsManager());
 }
 {code}
 and in AppBlock#render below check may fail(Need to test and confirm)
 {code}
if (callerUGI != null
  !(this.aclsManager.checkAccess(callerUGI,
 ApplicationAccessType.VIEW_APP, app.getUser(), appID) ||
  this.queueACLsManager.checkAccess(callerUGI,
 QueueACL.ADMINISTER_QUEUE, app.getQueue( {
   puts(You (User  + remoteUser
   + ) are not authorized to view application  + appID);
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1156) Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230881#comment-14230881
 ] 

Hadoop QA commented on YARN-1156:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12684530/YARN-1156.3.patch
  against trunk revision a7fba0b.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5967//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-YARN-Build/5967//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5967//console

This message is automatically generated.

 Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values
 -

 Key: YARN-1156
 URL: https://issues.apache.org/jira/browse/YARN-1156
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.1.0-beta
Reporter: Akira AJISAKA
Assignee: Tsuyoshi OZAWA
Priority: Minor
  Labels: metrics, newbie
 Fix For: 2.7.0

 Attachments: YARN-1156.1.patch, YARN-1156.2.patch, YARN-1156.3.patch


 AllocatedGB and AvailableGB metrics are now integer type. If there are four 
 times 500MB memory allocation to container, AllocatedGB is incremented four 
 times by {{(int)500/1024}}, which means 0. That is, the memory size allocated 
 is actually 2000MB, but the metrics shows 0GB. Let's use float type for these 
 metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting

2014-12-01 Thread Zhijie Shen (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230882#comment-14230882
 ] 

Zhijie Shen commented on YARN-2837:
---

bq. One question is, are we planning to support more types of status 
information in the future?

Timeline server is the stateless machine. Currently, the only state I think of 
now is the delegation token. Maybe in future, if we use queue to buffer the 
requests to process, we need to store those too when restarting the timeline 
server. IAC, let's leave it later.

bq. Shall we further restrict its visibility to YARN only (by 
{{@LimitedPrivate}})?

AFAIK, {{@Private}} equals to {{@LimitedPrivate(current project)}}, while 
{{@LimitedPrivate}} can mark a class private to others and multiple sub 
projects.

 Timeline server needs to recover the timeline DT when restarting
 

 Key: YARN-2837
 URL: https://issues.apache.org/jira/browse/YARN-2837
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
 Attachments: YARN-2837.1.patch, YARN-2837.2.patch


 Timeline server needs to recover the stateful information when restarting as 
 RM/NM/JHS does now. So far the stateful information only includes the 
 timeline DT. Without recovery, the timeline DT of the existing YARN apps is 
 not long valid, and cannot be renewed any more after the timeline server is 
 restarted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so

2014-12-01 Thread Zhijie Shen (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230890#comment-14230890
 ] 

Zhijie Shen commented on YARN-2890:
---

+1 Will commit the patch

 MiniMRYarnCluster should turn on timeline service if configured to do so
 

 Key: YARN-2890
 URL: https://issues.apache.org/jira/browse/YARN-2890
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mit Desai
Assignee: Mit Desai
 Fix For: 2.7.0

 Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, 
 YARN-2890.patch


 Currently the MiniMRYarnCluster does not consider the configuration value for 
 enabling timeline service before starting. The MiniYarnCluster should only 
 start the timeline service if it is configured to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so

2014-12-01 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230904#comment-14230904
 ] 

Hudson commented on YARN-2890:
--

FAILURE: Integrated in Hadoop-trunk-Commit #6623 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/6623/])
YARN-2890. MiniYARNCluster should start the timeline server based on the 
configuration. Contributed by Mit Desai. (zjshen: rev 
51af8d367de94689770f57c64bea3b244d7755f6)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
* hadoop-yarn-project/CHANGES.txt
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java


 MiniMRYarnCluster should turn on timeline service if configured to do so
 

 Key: YARN-2890
 URL: https://issues.apache.org/jira/browse/YARN-2890
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mit Desai
Assignee: Mit Desai
 Fix For: 2.7.0

 Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, 
 YARN-2890.patch


 Currently the MiniMRYarnCluster does not consider the configuration value for 
 enabling timeline service before starting. The MiniYarnCluster should only 
 start the timeline service if it is configured to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230915#comment-14230915
 ] 

Hadoop QA commented on YARN-2837:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12684536/YARN-2837.2.patch
  against trunk revision a7fba0b.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5968//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-YARN-Build/5968//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5968//console

This message is automatically generated.

 Timeline server needs to recover the timeline DT when restarting
 

 Key: YARN-2837
 URL: https://issues.apache.org/jira/browse/YARN-2837
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
 Fix For: 2.7.0

 Attachments: YARN-2837.1.patch, YARN-2837.2.patch


 Timeline server needs to recover the stateful information when restarting as 
 RM/NM/JHS does now. So far the stateful information only includes the 
 timeline DT. Without recovery, the timeline DT of the existing YARN apps is 
 not long valid, and cannot be renewed any more after the timeline server is 
 restarted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2894) When ACL's are enabled, if RM switches then application can not be viewed from web.

2014-12-01 Thread Rohith (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230919#comment-14230919
 ] 

Rohith commented on YARN-2894:
--

The changes are very big for getting RMContext from rm object. Given for 
future-proof, I will change it.

 When ACL's are enabled, if RM switches then application can not be viewed 
 from web.
 ---

 Key: YARN-2894
 URL: https://issues.apache.org/jira/browse/YARN-2894
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
 Fix For: 2.7.0

 Attachments: YARN-2894.patch


 Binding aclManager to RMWebApp would cause problem if RM is switched. There 
 could be some validation check may fail.
 I think , we should not bind aclManager for RMWebApp, instead we should get 
 from RM instance.
 In RMWebApp,
 {code}
 if (rm != null) {
   bind(ResourceManager.class).toInstance(rm);
   bind(RMContext.class).toInstance(rm.getRMContext());
   bind(ApplicationACLsManager.class).toInstance(
   rm.getApplicationACLsManager());
   bind(QueueACLsManager.class).toInstance(rm.getQueueACLsManager());
 }
 {code}
 and in AppBlock#render below check may fail(Need to test and confirm)
 {code}
if (callerUGI != null
  !(this.aclsManager.checkAccess(callerUGI,
 ApplicationAccessType.VIEW_APP, app.getUser(), appID) ||
  this.queueACLsManager.checkAccess(callerUGI,
 QueueACL.ADMINISTER_QUEUE, app.getQueue( {
   puts(You (User  + remoteUser
   + ) are not authorized to view application  + appID);
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (YARN-2837) Timeline server needs to recover the timeline DT when restarting

2014-12-01 Thread Zhijie Shen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated YARN-2837:
--
Comment: was deleted

(was: Committed to trunk and branch-2. Thanks, [~mitdesai]!)

 Timeline server needs to recover the timeline DT when restarting
 

 Key: YARN-2837
 URL: https://issues.apache.org/jira/browse/YARN-2837
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
 Fix For: 2.7.0

 Attachments: YARN-2837.1.patch, YARN-2837.2.patch


 Timeline server needs to recover the stateful information when restarting as 
 RM/NM/JHS does now. So far the stateful information only includes the 
 timeline DT. Without recovery, the timeline DT of the existing YARN apps is 
 not long valid, and cannot be renewed any more after the timeline server is 
 restarted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (YARN-2837) Timeline server needs to recover the timeline DT when restarting

2014-12-01 Thread Zhijie Shen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen reopened YARN-2837:
---

Wrong operation on this Jira. Reopen it.

 Timeline server needs to recover the timeline DT when restarting
 

 Key: YARN-2837
 URL: https://issues.apache.org/jira/browse/YARN-2837
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
 Fix For: 2.7.0

 Attachments: YARN-2837.1.patch, YARN-2837.2.patch


 Timeline server needs to recover the stateful information when restarting as 
 RM/NM/JHS does now. So far the stateful information only includes the 
 timeline DT. Without recovery, the timeline DT of the existing YARN apps is 
 not long valid, and cannot be renewed any more after the timeline server is 
 restarted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2837) Timeline server needs to recover the timeline DT when restarting

2014-12-01 Thread Zhijie Shen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated YARN-2837:
--
Attachment: YARN-2837.3.patch

Create a new patch to suppress the findbugs warning. The reason is that 
KeyParser will only be used internally, and I just copy and paste it to 
refactor the code. While deep copying may solve the warning, it will 
unnecessarily downgrade the performance.


 Timeline server needs to recover the timeline DT when restarting
 

 Key: YARN-2837
 URL: https://issues.apache.org/jira/browse/YARN-2837
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
 Fix For: 2.7.0

 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch


 Timeline server needs to recover the stateful information when restarting as 
 RM/NM/JHS does now. So far the stateful information only includes the 
 timeline DT. Without recovery, the timeline DT of the existing YARN apps is 
 not long valid, and cannot be renewed any more after the timeline server is 
 restarted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2894) When ACL's are enabled, if RM switches then application can not be viewed from web.

2014-12-01 Thread Jian He (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230986#comment-14230986
 ] 

Jian He commented on YARN-2894:
---

bq. The changes are very big for getting RMContext from rm object
How big that can be ? I'm fine to stay as is, if that's too big change.

 When ACL's are enabled, if RM switches then application can not be viewed 
 from web.
 ---

 Key: YARN-2894
 URL: https://issues.apache.org/jira/browse/YARN-2894
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
 Fix For: 2.7.0

 Attachments: YARN-2894.patch


 Binding aclManager to RMWebApp would cause problem if RM is switched. There 
 could be some validation check may fail.
 I think , we should not bind aclManager for RMWebApp, instead we should get 
 from RM instance.
 In RMWebApp,
 {code}
 if (rm != null) {
   bind(ResourceManager.class).toInstance(rm);
   bind(RMContext.class).toInstance(rm.getRMContext());
   bind(ApplicationACLsManager.class).toInstance(
   rm.getApplicationACLsManager());
   bind(QueueACLsManager.class).toInstance(rm.getQueueACLsManager());
 }
 {code}
 and in AppBlock#render below check may fail(Need to test and confirm)
 {code}
if (callerUGI != null
  !(this.aclsManager.checkAccess(callerUGI,
 ApplicationAccessType.VIEW_APP, app.getUser(), appID) ||
  this.queueACLsManager.checkAccess(callerUGI,
 QueueACL.ADMINISTER_QUEUE, app.getQueue( {
   puts(You (User  + remoteUser
   + ) are not authorized to view application  + appID);
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-1156) Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values

2014-12-01 Thread Tsuyoshi OZAWA (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated YARN-1156:
-
Attachment: YARN-1156.4.patch

Fixing a warning by findbugs.

 Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values
 -

 Key: YARN-1156
 URL: https://issues.apache.org/jira/browse/YARN-1156
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.1.0-beta
Reporter: Akira AJISAKA
Assignee: Tsuyoshi OZAWA
Priority: Minor
  Labels: metrics, newbie
 Fix For: 2.7.0

 Attachments: YARN-1156.1.patch, YARN-1156.2.patch, YARN-1156.3.patch, 
 YARN-1156.4.patch


 AllocatedGB and AvailableGB metrics are now integer type. If there are four 
 times 500MB memory allocation to container, AllocatedGB is incremented four 
 times by {{(int)500/1024}}, which means 0. That is, the memory size allocated 
 is actually 2000MB, but the metrics shows 0GB. Let's use float type for these 
 metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2894) When ACL's are enabled, if RM switches then application can not be viewed from web.

2014-12-01 Thread Rohith (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231016#comment-14231016
 ] 

Rohith commented on YARN-2894:
--

It is around 18 classes to be changed. However I have done changes. Will upload 
patch soon

 When ACL's are enabled, if RM switches then application can not be viewed 
 from web.
 ---

 Key: YARN-2894
 URL: https://issues.apache.org/jira/browse/YARN-2894
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
 Fix For: 2.7.0

 Attachments: YARN-2894.patch


 Binding aclManager to RMWebApp would cause problem if RM is switched. There 
 could be some validation check may fail.
 I think , we should not bind aclManager for RMWebApp, instead we should get 
 from RM instance.
 In RMWebApp,
 {code}
 if (rm != null) {
   bind(ResourceManager.class).toInstance(rm);
   bind(RMContext.class).toInstance(rm.getRMContext());
   bind(ApplicationACLsManager.class).toInstance(
   rm.getApplicationACLsManager());
   bind(QueueACLsManager.class).toInstance(rm.getQueueACLsManager());
 }
 {code}
 and in AppBlock#render below check may fail(Need to test and confirm)
 {code}
if (callerUGI != null
  !(this.aclsManager.checkAccess(callerUGI,
 ApplicationAccessType.VIEW_APP, app.getUser(), appID) ||
  this.queueACLsManager.checkAccess(callerUGI,
 QueueACL.ADMINISTER_QUEUE, app.getQueue( {
   puts(You (User  + remoteUser
   + ) are not authorized to view application  + appID);
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2894) When ACL's are enabled, if RM switches then application can not be viewed from web.

2014-12-01 Thread Rohith (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith updated YARN-2894:
-
Attachment: YARN-2894.1.patch

 When ACL's are enabled, if RM switches then application can not be viewed 
 from web.
 ---

 Key: YARN-2894
 URL: https://issues.apache.org/jira/browse/YARN-2894
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
 Fix For: 2.7.0

 Attachments: YARN-2894.1.patch, YARN-2894.patch


 Binding aclManager to RMWebApp would cause problem if RM is switched. There 
 could be some validation check may fail.
 I think , we should not bind aclManager for RMWebApp, instead we should get 
 from RM instance.
 In RMWebApp,
 {code}
 if (rm != null) {
   bind(ResourceManager.class).toInstance(rm);
   bind(RMContext.class).toInstance(rm.getRMContext());
   bind(ApplicationACLsManager.class).toInstance(
   rm.getApplicationACLsManager());
   bind(QueueACLsManager.class).toInstance(rm.getQueueACLsManager());
 }
 {code}
 and in AppBlock#render below check may fail(Need to test and confirm)
 {code}
if (callerUGI != null
  !(this.aclsManager.checkAccess(callerUGI,
 ApplicationAccessType.VIEW_APP, app.getUser(), appID) ||
  this.queueACLsManager.checkAccess(callerUGI,
 QueueACL.ADMINISTER_QUEUE, app.getQueue( {
   puts(You (User  + remoteUser
   + ) are not authorized to view application  + appID);
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2894) When ACL's are enabled, if RM switches then application can not be viewed from web.

2014-12-01 Thread Rohith (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231028#comment-14231028
 ] 

Rohith commented on YARN-2894:
--

Updated the patch with following changes.
# Removed the binding of RMContext ,aclManager and queueAclManager in RMWebApp 
and other test cases.
# Did clean up where both RM instance and RMContext instance were passed to 
{{MetricsOverviewTable}} {{ClusterMetricsInfo}} {{UserMetricsInfo}} classes 
which never used. I do not think it breaks compatability. What do you think?
# I ran test locally which all are passing. But many tests failed in previous 
QA result because of reason address bind exception. I am analyzing it, will 
fix in another jira.

 When ACL's are enabled, if RM switches then application can not be viewed 
 from web.
 ---

 Key: YARN-2894
 URL: https://issues.apache.org/jira/browse/YARN-2894
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
 Fix For: 2.7.0

 Attachments: YARN-2894.1.patch, YARN-2894.patch


 Binding aclManager to RMWebApp would cause problem if RM is switched. There 
 could be some validation check may fail.
 I think , we should not bind aclManager for RMWebApp, instead we should get 
 from RM instance.
 In RMWebApp,
 {code}
 if (rm != null) {
   bind(ResourceManager.class).toInstance(rm);
   bind(RMContext.class).toInstance(rm.getRMContext());
   bind(ApplicationACLsManager.class).toInstance(
   rm.getApplicationACLsManager());
   bind(QueueACLsManager.class).toInstance(rm.getQueueACLsManager());
 }
 {code}
 and in AppBlock#render below check may fail(Need to test and confirm)
 {code}
if (callerUGI != null
  !(this.aclsManager.checkAccess(callerUGI,
 ApplicationAccessType.VIEW_APP, app.getUser(), appID) ||
  this.queueACLsManager.checkAccess(callerUGI,
 QueueACL.ADMINISTER_QUEUE, app.getQueue( {
   puts(You (User  + remoteUser
   + ) are not authorized to view application  + appID);
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-1156) Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231038#comment-14231038
 ] 

Hadoop QA commented on YARN-1156:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12684561/YARN-1156.4.patch
  against trunk revision 51af8d3.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5969//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5969//console

This message is automatically generated.

 Change NodeManager AllocatedGB and AvailableGB metrics to show decimal values
 -

 Key: YARN-1156
 URL: https://issues.apache.org/jira/browse/YARN-1156
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.1.0-beta
Reporter: Akira AJISAKA
Assignee: Tsuyoshi OZAWA
Priority: Minor
  Labels: metrics, newbie
 Fix For: 2.7.0

 Attachments: YARN-1156.1.patch, YARN-1156.2.patch, YARN-1156.3.patch, 
 YARN-1156.4.patch


 AllocatedGB and AvailableGB metrics are now integer type. If there are four 
 times 500MB memory allocation to container, AllocatedGB is incremented four 
 times by {{(int)500/1024}}, which means 0. That is, the memory size allocated 
 is actually 2000MB, but the metrics shows 0GB. Let's use float type for these 
 metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2190) Provide a Windows container executor that can limit memory and CPU

2014-12-01 Thread Remus Rusanu (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231073#comment-14231073
 ] 

Remus Rusanu commented on YARN-2190:


{
+#ifdef NTDDI_WIN8
}

I think you need a runtime version check too. Is possible to compile against 
Win8 SDK but run on lower version.

 Provide a Windows container executor that can limit memory and CPU
 --

 Key: YARN-2190
 URL: https://issues.apache.org/jira/browse/YARN-2190
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: nodemanager
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, 
 YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch, 
 YARN-2190.6.patch, YARN-2190.7.patch


 Yarn default container executor on Windows does not set the resource limit on 
 the containers currently. The memory limit is enforced by a separate 
 monitoring thread. The container implementation on Windows uses Job Object 
 right now. The latest Windows (8 or later) API allows CPU and memory limits 
 on the job objects. We want to create a Windows container executor that sets 
 the limits on job objects thus provides resource enforcement at OS level.
 http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2894) When ACL's are enabled, if RM switches then application can not be viewed from web.

2014-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231076#comment-14231076
 ] 

Hadoop QA commented on YARN-2894:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12684562/YARN-2894.1.patch
  against trunk revision 9fa2990.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 8 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/5970//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5970//console

This message is automatically generated.

 When ACL's are enabled, if RM switches then application can not be viewed 
 from web.
 ---

 Key: YARN-2894
 URL: https://issues.apache.org/jira/browse/YARN-2894
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Rohith
Assignee: Rohith
 Fix For: 2.7.0

 Attachments: YARN-2894.1.patch, YARN-2894.patch


 Binding aclManager to RMWebApp would cause problem if RM is switched. There 
 could be some validation check may fail.
 I think , we should not bind aclManager for RMWebApp, instead we should get 
 from RM instance.
 In RMWebApp,
 {code}
 if (rm != null) {
   bind(ResourceManager.class).toInstance(rm);
   bind(RMContext.class).toInstance(rm.getRMContext());
   bind(ApplicationACLsManager.class).toInstance(
   rm.getApplicationACLsManager());
   bind(QueueACLsManager.class).toInstance(rm.getQueueACLsManager());
 }
 {code}
 and in AppBlock#render below check may fail(Need to test and confirm)
 {code}
if (callerUGI != null
  !(this.aclsManager.checkAccess(callerUGI,
 ApplicationAccessType.VIEW_APP, app.getUser(), appID) ||
  this.queueACLsManager.checkAccess(callerUGI,
 QueueACL.ADMINISTER_QUEUE, app.getQueue( {
   puts(You (User  + remoteUser
   + ) are not authorized to view application  + appID);
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2880) Add a test in TestRMRestart to make sure node labels will be recovered if it is enabled

2014-12-01 Thread Rohith (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231089#comment-14231089
 ] 

Rohith commented on YARN-2880:
--

The reason for test failures are newly added test was setting explicitely 
RM_SCHEDULER!!! Since it is Parameterized on scheduler, test which runs after 
newly added test were failed. I will upload new patch with fix.!!

 Add a test in TestRMRestart to make sure node labels will be recovered if it 
 is enabled
 ---

 Key: YARN-2880
 URL: https://issues.apache.org/jira/browse/YARN-2880
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Wangda Tan
Assignee: Rohith
 Attachments: 0001-YARN-2880.patch, YARN-2880.1.patch


 As suggested by [~ozawa], 
 [link|https://issues.apache.org/jira/browse/YARN-2800?focusedCommentId=14217569page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14217569].
  We should have a such test to make sure there will be no regression



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2880) Add a test in TestRMRestart to make sure node labels will be recovered if it is enabled

2014-12-01 Thread Rohith (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith updated YARN-2880:
-
Attachment: YARN-2880.1.patch

 Add a test in TestRMRestart to make sure node labels will be recovered if it 
 is enabled
 ---

 Key: YARN-2880
 URL: https://issues.apache.org/jira/browse/YARN-2880
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Wangda Tan
Assignee: Rohith
 Attachments: 0001-YARN-2880.patch, YARN-2880.1.patch, 
 YARN-2880.1.patch


 As suggested by [~ozawa], 
 [link|https://issues.apache.org/jira/browse/YARN-2800?focusedCommentId=14217569page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14217569].
  We should have a such test to make sure there will be no regression



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2880) Add a test in TestRMRestart to make sure node labels will be recovered if it is enabled

2014-12-01 Thread Rohith (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith updated YARN-2880:
-
Attachment: YARN-2880.1.patch

 Add a test in TestRMRestart to make sure node labels will be recovered if it 
 is enabled
 ---

 Key: YARN-2880
 URL: https://issues.apache.org/jira/browse/YARN-2880
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Wangda Tan
Assignee: Rohith
 Attachments: 0001-YARN-2880.patch, YARN-2880.1.patch, 
 YARN-2880.1.patch, YARN-2880.1.patch


 As suggested by [~ozawa], 
 [link|https://issues.apache.org/jira/browse/YARN-2800?focusedCommentId=14217569page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14217569].
  We should have a such test to make sure there will be no regression



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2880) Add a test in TestRMRestart to make sure node labels will be recovered if it is enabled

2014-12-01 Thread Rohith (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231113#comment-14231113
 ] 

Rohith commented on YARN-2880:
--

Attached patch that  fixes test failures.

 Add a test in TestRMRestart to make sure node labels will be recovered if it 
 is enabled
 ---

 Key: YARN-2880
 URL: https://issues.apache.org/jira/browse/YARN-2880
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Wangda Tan
Assignee: Rohith
 Attachments: 0001-YARN-2880.patch, YARN-2880.1.patch, 
 YARN-2880.1.patch, YARN-2880.1.patch


 As suggested by [~ozawa], 
 [link|https://issues.apache.org/jira/browse/YARN-2800?focusedCommentId=14217569page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14217569].
  We should have a such test to make sure there will be no regression



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-2664) Improve RM webapp to expose info about reservations.

2014-12-01 Thread Matteo Mazzucchelli (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Mazzucchelli updated YARN-2664:
--
Attachment: YARN-2664.6.patch

The YARN-2664.6.patch correct the previous unit tests

 Improve RM webapp to expose info about reservations.
 

 Key: YARN-2664
 URL: https://issues.apache.org/jira/browse/YARN-2664
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Carlo Curino
Assignee: Matteo Mazzucchelli
 Attachments: PlannerPage_screenshot.pdf, YARN-2664.1.patch, 
 YARN-2664.2.patch, YARN-2664.3.patch, YARN-2664.4.patch, YARN-2664.5.patch, 
 YARN-2664.6.patch, YARN-2664.patch


 YARN-1051 provides a new functionality in the RM to ask for reservation on 
 resources. Exposing this through the webapp GUI is important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)