date:20130815

[
https://issues.apache.org/jira/browse/YARN-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740735#comment-13740735
]

Bikas Saha commented on YARN-1004:
--

IMO the intent of yarn.scheduler.max is to be an admin setting that restricts
how much resource can be given to any one container. Its exposed via public
YARN API. All schedulers are supposed to enforce the admin value and not
determine it for themselves. Hence I dont think it should be scheduler specific.
yarn.scheduler.min is scheduler internal logic on how it simplifies the
bin-packing problem. Both current schedulers use it and its not exposed by any
YARN API because its of no use to the user. We may split it to be scheduler
specific but do we really see either scheduler not using it in the foreseeable
future? Perhaps we are causing more grief than good by splitting them.
increment-allocation-mb is used only in the fair scheduler. lets just rename
that to be scheduler specific.

yarn.scheduler.minimum|maximum|increment-allocation-mb should be prefixed
with the scheduler type
--

Key: YARN-1004
URL: https://issues.apache.org/jira/browse/YARN-1004
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Priority: Blocker
Fix For: 2.1.0-beta

Attachments: YARN-1004.patch

As yarn.scheduler.minimum-allocation-mb is now a scheduler-specific
configuration, and functions differently for the Fair and Capacity
schedulers, it would be less confusing for the config names to include the
scheduler names, i.e. yarn.scheduler.fair.minimum-allocation-mb,
yarn.scheduler.capacity.minimum-allocation-mb, and
yarn.scheduler.fifo.minimum-allocation-mb.
The same goes for yarn.scheduler.increment-allocation-mb, which only exists
for the Fair Scheduler, and yarn.scheduler.maximum-allocation-mb, for
consistency.
If we wish to preserve backwards compatibility, we can deprecate the old
configs to the new ones.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1044) used/min/max resources do not display info in the scheduler page


 [ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-1044:
--

Attachment: (was: yarn-1044.patch)

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1044) used/min/max resources do not display info in the scheduler page


 [ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-1044:
--

Attachment: yarn-1044.patch

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740745#comment-13740745
 ] 

Hadoop QA commented on YARN-1044:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12598160/yarn-1044.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1721//console

This message is automatically generated.

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1044) used/min/max resources do not display info in the scheduler page


 [ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-1044:
--

Attachment: (was: yarn-1044.patch)

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1044) used/min/max resources do not display info in the scheduler page


 [ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-1044:
--

Attachment: yarn-1044.patch

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740752#comment-13740752
 ] 

Hadoop QA commented on YARN-1044:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12598163/yarn-1044.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1722//console

This message is automatically generated.

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740774#comment-13740774
 ] 

Hadoop QA commented on YARN-1044:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12598163/yarn-1044.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1723//console

This message is automatically generated.

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1045) Improve toString implementation for PBImpls

2013-08-15 Thread Siddharth Seth (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated YARN-1045:
-

Hadoop Flags: Reviewed

 Improve toString implementation for PBImpls
 ---

 Key: YARN-1045
 URL: https://issues.apache.org/jira/browse/YARN-1045
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Jian He
 Fix For: 2.1.1-beta

 Attachments: YARN-1045.1.patch, YARN-1045.patch


 The generic toString implementation that is used in most of the PBImpls 
 {code}getProto().toString().replaceAll(\\n, , ).replaceAll(\\s+,  
 );{code} is rather inefficient - replacing \n and \s to generate a one 
 line string. Instead, we can use 
 {code}TextFormat.shortDebugString(getProto());{code}.
 If we can get this into 2.1.0 - great, otherwise the next release.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1045) Improve toString implementation for PBImpls


[ 
https://issues.apache.org/jira/browse/YARN-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740780#comment-13740780
 ] 

Hudson commented on YARN-1045:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #4267 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/4267/])
YARN-1045. Improve toString implementation for PBImpls. Contributed by Jian He. 
(sseth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514185)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/CancelDelegationTokenRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/CancelDelegationTokenResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/FinishApplicationMasterRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/FinishApplicationMasterResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationReportRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationReportResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterMetricsRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterMetricsResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodesRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodesResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetContainerStatusesRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetContainerStatusesResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetDelegationTokenRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetDelegationTokenResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNewApplicationRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNewApplicationResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueInfoRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueInfoResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueUserAclsInfoRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueUserAclsInfoResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/KillApplicationRequestPBImpl.java
*

[jira] [Commented] (YARN-337) RM handles killed application tracking URL poorly


[ 
https://issues.apache.org/jira/browse/YARN-337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740868#comment-13740868
 ] 

Hudson commented on YARN-337:
-

SUCCESS: Integrated in Hadoop-Yarn-trunk #302 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/302/])
YARN-337. RM handles killed application tracking URL poorly. Contributed by 
Jason Lowe (jlowe: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513888)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java


 RM handles killed application tracking URL poorly
 -

 Key: YARN-337
 URL: https://issues.apache.org/jira/browse/YARN-337
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.0.2-alpha, 0.23.5
Reporter: Jason Lowe
Assignee: Jason Lowe
  Labels: usability
 Fix For: 3.0.0, 0.23.10, 2.1.1-beta

 Attachments: YARN-337.patch


 When the ResourceManager kills an application, it leaves the proxy URL 
 redirecting to the original tracking URL for the application even though the 
 ApplicationMaster is no longer there to service it.  It should redirect it 
 somewhere more useful, like the RM's web page for the application, where the 
 user can find that the application was killed and links to the AM logs.
 In addition, sometimes the AM during teardown from the kill can attempt to 
 unregister and provide an updated tracking URL, but unfortunately the RM has 
 forgotten the AM due to the kill and refuses to process the unregistration. 
  Instead it logs:
 {noformat}
 2013-01-09 17:37:49,671 [IPC Server handler 2 on 8030] ERROR
 org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: 
 AppAttemptId doesnt exist in cache appattempt_1357575694478_28614_01
 {noformat}
 It should go ahead and process the unregistration to update the tracking URL 
 since the application offered it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1056) Fix configs yarn.resourcemanager.resourcemanager.connect.{max.wait.secs|retry_interval.secs}


[ 
https://issues.apache.org/jira/browse/YARN-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740870#comment-13740870
 ] 

Hudson commented on YARN-1056:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #302 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/302/])
YARN-1056. Remove dual use of string 'resourcemanager' in 
yarn.resourcemanager.connect.{max.wait.secs|retry_interval.secs}. Contributed 
by Karthik Kambatla. (acmurthy: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514135)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java


 Fix configs 
 yarn.resourcemanager.resourcemanager.connect.{max.wait.secs|retry_interval.secs}
 

 Key: YARN-1056
 URL: https://issues.apache.org/jira/browse/YARN-1056
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Trivial
  Labels: conf
 Fix For: 2.1.0-beta

 Attachments: yarn-1056-1.patch, yarn-1056-1.patch, yarn-1056-2.patch


 Fix configs 
 yarn.resourcemanager.resourcemanager.connect.{max.wait.secs|retry_interval.secs}
  to have a *resourcemanager* only once, make them consistent with other such 
 yarn configs and add entries in yarn-default.xml

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1045) Improve toString implementation for PBImpls


[ 
https://issues.apache.org/jira/browse/YARN-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740867#comment-13740867
 ] 

Hudson commented on YARN-1045:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #302 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/302/])
YARN-1045. Improve toString implementation for PBImpls. Contributed by Jian He. 
(sseth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514185)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/CancelDelegationTokenRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/CancelDelegationTokenResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/FinishApplicationMasterRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/FinishApplicationMasterResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationReportRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationReportResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterMetricsRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterMetricsResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodesRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodesResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetContainerStatusesRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetContainerStatusesResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetDelegationTokenRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetDelegationTokenResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNewApplicationRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNewApplicationResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueInfoRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueInfoResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueUserAclsInfoRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueUserAclsInfoResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/KillApplicationRequestPBImpl.java
*

[jira] [Commented] (YARN-292) ResourceManager throws ArrayIndexOutOfBoundsException while handling CONTAINER_ALLOCATED for application attempt

2013-08-15 Thread nemon lou (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740880#comment-13740880
 ] 

nemon lou commented on YARN-292:


FIFO Scheduler uses TreeMap to keep applications in FIFO 
order,ConcurrentHashMap will break this featrue. Right?

 ResourceManager throws ArrayIndexOutOfBoundsException while handling 
 CONTAINER_ALLOCATED for application attempt
 

 Key: YARN-292
 URL: https://issues.apache.org/jira/browse/YARN-292
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Affects Versions: 2.0.1-alpha
Reporter: Devaraj K
Assignee: Zhijie Shen
 Attachments: YARN-292.1.patch, YARN-292.2.patch


 {code:xml}
 2012-12-26 08:41:15,030 ERROR 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler: 
 Calling allocate on removed or non existant application 
 appattempt_1356385141279_49525_01
 2012-12-26 08:41:15,031 ERROR 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type CONTAINER_ALLOCATED for applicationAttempt 
 application_1356385141279_49525
 java.lang.ArrayIndexOutOfBoundsException: 0
   at java.util.Arrays$ArrayList.get(Arrays.java:3381)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AMContainerAllocatedTransition.transition(RMAppAttemptImpl.java:655)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AMContainerAllocatedTransition.transition(RMAppAttemptImpl.java:644)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:490)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:80)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:433)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:414)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
   at java.lang.Thread.run(Thread.java:662)
  {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-337) RM handles killed application tracking URL poorly


[ 
https://issues.apache.org/jira/browse/YARN-337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740898#comment-13740898
 ] 

Hudson commented on YARN-337:
-

FAILURE: Integrated in Hadoop-Hdfs-0.23-Build #700 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/700/])
svn merge -c 1513888 FIXES: YARN-337. RM handles killed application tracking 
URL poorly. Contributed by Jason Lowe (jlowe: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513894)
* /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* 
/hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java


 RM handles killed application tracking URL poorly
 -

 Key: YARN-337
 URL: https://issues.apache.org/jira/browse/YARN-337
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.0.2-alpha, 0.23.5
Reporter: Jason Lowe
Assignee: Jason Lowe
  Labels: usability
 Fix For: 3.0.0, 0.23.10, 2.1.1-beta

 Attachments: YARN-337.patch


 When the ResourceManager kills an application, it leaves the proxy URL 
 redirecting to the original tracking URL for the application even though the 
 ApplicationMaster is no longer there to service it.  It should redirect it 
 somewhere more useful, like the RM's web page for the application, where the 
 user can find that the application was killed and links to the AM logs.
 In addition, sometimes the AM during teardown from the kill can attempt to 
 unregister and provide an updated tracking URL, but unfortunately the RM has 
 forgotten the AM due to the kill and refuses to process the unregistration. 
  Instead it logs:
 {noformat}
 2013-01-09 17:37:49,671 [IPC Server handler 2 on 8030] ERROR
 org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: 
 AppAttemptId doesnt exist in cache appattempt_1357575694478_28614_01
 {noformat}
 It should go ahead and process the unregistration to update the tracking URL 
 since the application offered it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1004) yarn.scheduler.minimum|maximum|increment-allocation-mb should be prefixed with the scheduler type

2013-08-15 Thread Alejandro Abdelnur (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740917#comment-13740917
 ] 

Alejandro Abdelnur commented on YARN-1004:
--

My motivation for this JIRA was that the *incr* properties are not know for 
capacity and fifo and that *min* has a different meaning for fair than for 
capacity and fifo.

If others think this is a non issue and will cause more confusion than good, 
I'm OK with closing this as won't fix.


 yarn.scheduler.minimum|maximum|increment-allocation-mb should be prefixed 
 with the scheduler type 
 --

 Key: YARN-1004
 URL: https://issues.apache.org/jira/browse/YARN-1004
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: scheduler
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Priority: Blocker
 Fix For: 2.1.0-beta

 Attachments: YARN-1004.patch


 As yarn.scheduler.minimum-allocation-mb is now a scheduler-specific 
 configuration, and functions differently for the Fair and Capacity 
 schedulers, it would be less confusing for the config names to include the 
 scheduler names, i.e. yarn.scheduler.fair.minimum-allocation-mb, 
 yarn.scheduler.capacity.minimum-allocation-mb, and 
 yarn.scheduler.fifo.minimum-allocation-mb.
 The same goes for yarn.scheduler.increment-allocation-mb, which only exists 
 for the Fair Scheduler, and yarn.scheduler.maximum-allocation-mb, for 
 consistency.
 If we wish to preserve backwards compatibility, we can deprecate the old 
 configs to the new ones. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-337) RM handles killed application tracking URL poorly


[ 
https://issues.apache.org/jira/browse/YARN-337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740940#comment-13740940
 ] 

Hudson commented on YARN-337:
-

FAILURE: Integrated in Hadoop-Hdfs-trunk #1492 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1492/])
YARN-337. RM handles killed application tracking URL poorly. Contributed by 
Jason Lowe (jlowe: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513888)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java


 RM handles killed application tracking URL poorly
 -

 Key: YARN-337
 URL: https://issues.apache.org/jira/browse/YARN-337
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.0.2-alpha, 0.23.5
Reporter: Jason Lowe
Assignee: Jason Lowe
  Labels: usability
 Fix For: 3.0.0, 0.23.10, 2.1.1-beta

 Attachments: YARN-337.patch


 When the ResourceManager kills an application, it leaves the proxy URL 
 redirecting to the original tracking URL for the application even though the 
 ApplicationMaster is no longer there to service it.  It should redirect it 
 somewhere more useful, like the RM's web page for the application, where the 
 user can find that the application was killed and links to the AM logs.
 In addition, sometimes the AM during teardown from the kill can attempt to 
 unregister and provide an updated tracking URL, but unfortunately the RM has 
 forgotten the AM due to the kill and refuses to process the unregistration. 
  Instead it logs:
 {noformat}
 2013-01-09 17:37:49,671 [IPC Server handler 2 on 8030] ERROR
 org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: 
 AppAttemptId doesnt exist in cache appattempt_1357575694478_28614_01
 {noformat}
 It should go ahead and process the unregistration to update the tracking URL 
 since the application offered it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1045) Improve toString implementation for PBImpls


[ 
https://issues.apache.org/jira/browse/YARN-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740939#comment-13740939
 ] 

Hudson commented on YARN-1045:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #1492 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1492/])
YARN-1045. Improve toString implementation for PBImpls. Contributed by Jian He. 
(sseth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514185)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/CancelDelegationTokenRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/CancelDelegationTokenResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/FinishApplicationMasterRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/FinishApplicationMasterResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationReportRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationReportResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterMetricsRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterMetricsResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodesRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodesResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetContainerStatusesRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetContainerStatusesResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetDelegationTokenRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetDelegationTokenResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNewApplicationRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNewApplicationResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueInfoRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueInfoResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueUserAclsInfoRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueUserAclsInfoResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/KillApplicationRequestPBImpl.java
*

[jira] [Commented] (YARN-1056) Fix configs yarn.resourcemanager.resourcemanager.connect.{max.wait.secs|retry_interval.secs}


[ 
https://issues.apache.org/jira/browse/YARN-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740942#comment-13740942
 ] 

Hudson commented on YARN-1056:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #1492 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1492/])
YARN-1056. Remove dual use of string 'resourcemanager' in 
yarn.resourcemanager.connect.{max.wait.secs|retry_interval.secs}. Contributed 
by Karthik Kambatla. (acmurthy: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514135)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java


 Fix configs 
 yarn.resourcemanager.resourcemanager.connect.{max.wait.secs|retry_interval.secs}
 

 Key: YARN-1056
 URL: https://issues.apache.org/jira/browse/YARN-1056
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Trivial
  Labels: conf
 Fix For: 2.1.0-beta

 Attachments: yarn-1056-1.patch, yarn-1056-1.patch, yarn-1056-2.patch


 Fix configs 
 yarn.resourcemanager.resourcemanager.connect.{max.wait.secs|retry_interval.secs}
  to have a *resourcemanager* only once, make them consistent with other such 
 yarn configs and add entries in yarn-default.xml

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1045) Improve toString implementation for PBImpls


[ 
https://issues.apache.org/jira/browse/YARN-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741010#comment-13741010
 ] 

Hudson commented on YARN-1045:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1519 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1519/])
YARN-1045. Improve toString implementation for PBImpls. Contributed by Jian He. 
(sseth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514185)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/CancelDelegationTokenRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/CancelDelegationTokenResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/FinishApplicationMasterRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/FinishApplicationMasterResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationReportRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationReportResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterMetricsRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterMetricsResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodesRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodesResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetContainerStatusesRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetContainerStatusesResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetDelegationTokenRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetDelegationTokenResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNewApplicationRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetNewApplicationResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueInfoRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueInfoResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueUserAclsInfoRequestPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueUserAclsInfoResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/KillApplicationRequestPBImpl.java
*

[jira] [Commented] (YARN-337) RM handles killed application tracking URL poorly


[ 
https://issues.apache.org/jira/browse/YARN-337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741011#comment-13741011
 ] 

Hudson commented on YARN-337:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1519 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1519/])
YARN-337. RM handles killed application tracking URL poorly. Contributed by 
Jason Lowe (jlowe: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1513888)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java


 RM handles killed application tracking URL poorly
 -

 Key: YARN-337
 URL: https://issues.apache.org/jira/browse/YARN-337
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.0.2-alpha, 0.23.5
Reporter: Jason Lowe
Assignee: Jason Lowe
  Labels: usability
 Fix For: 3.0.0, 0.23.10, 2.1.1-beta

 Attachments: YARN-337.patch


 When the ResourceManager kills an application, it leaves the proxy URL 
 redirecting to the original tracking URL for the application even though the 
 ApplicationMaster is no longer there to service it.  It should redirect it 
 somewhere more useful, like the RM's web page for the application, where the 
 user can find that the application was killed and links to the AM logs.
 In addition, sometimes the AM during teardown from the kill can attempt to 
 unregister and provide an updated tracking URL, but unfortunately the RM has 
 forgotten the AM due to the kill and refuses to process the unregistration. 
  Instead it logs:
 {noformat}
 2013-01-09 17:37:49,671 [IPC Server handler 2 on 8030] ERROR
 org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: 
 AppAttemptId doesnt exist in cache appattempt_1357575694478_28614_01
 {noformat}
 It should go ahead and process the unregistration to update the tracking URL 
 since the application offered it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1056) Fix configs yarn.resourcemanager.resourcemanager.connect.{max.wait.secs|retry_interval.secs}


[ 
https://issues.apache.org/jira/browse/YARN-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741013#comment-13741013
 ] 

Hudson commented on YARN-1056:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1519 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1519/])
YARN-1056. Remove dual use of string 'resourcemanager' in 
yarn.resourcemanager.connect.{max.wait.secs|retry_interval.secs}. Contributed 
by Karthik Kambatla. (acmurthy: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514135)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java


 Fix configs 
 yarn.resourcemanager.resourcemanager.connect.{max.wait.secs|retry_interval.secs}
 

 Key: YARN-1056
 URL: https://issues.apache.org/jira/browse/YARN-1056
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Trivial
  Labels: conf
 Fix For: 2.1.0-beta

 Attachments: yarn-1056-1.patch, yarn-1056-1.patch, yarn-1056-2.patch


 Fix configs 
 yarn.resourcemanager.resourcemanager.connect.{max.wait.secs|retry_interval.secs}
  to have a *resourcemanager* only once, make them consistent with other such 
 yarn configs and add entries in yarn-default.xml

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1024) Define a virtual core unambigiously

2013-08-15 Thread Arun C Murthy (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741051#comment-13741051
]

Arun C Murthy commented on YARN-1024:
-

[~sandyr] Thanks for taking the time to clearly elucidate our long
always-half-confused discussion i.e. the longwindedness! I think we could be
close to a solution here, I really do - though, I'm not betting my house yet.
*smile*

To paraphrase your proposal (mainly for my own benefit):
# Split current (get,set)VirtualCores into (get,set)YCUPerCore and
(get,set)Cores.
# There is a cluster-wide constant of maxYCUPerCore
# The schedulers use {{core * YCUPerCore}} to do resource-allocation
comparisons.

The one issue that we need to think about is that we'll need to enhance the
schedulers to track how much YCUs are available on which core on any given
node... you could have 5 YCUs in a node but split 3-2-1 across 3 cores. Any
good ideas on how to get to this?

Define a virtual core unambigiously
---

Key: YARN-1024
URL: https://issues.apache.org/jira/browse/YARN-1024
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Arun C Murthy

We need to clearly define the meaning of a virtual core unambiguously so that
it's easy to migrate applications between clusters.
For e.g. here is Amazon EC2 definition of ECU:
http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it
Essentially we need to clearly define a YARN Virtual Core (YVC).
Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the
equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.*

[jira] [Commented] (YARN-1024) Define a virtual core unambigiously

2013-08-15 Thread Alejandro Abdelnur (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741081#comment-13741081
 ] 

Alejandro Abdelnur commented on YARN-1024:
--

Instead cores, could we talk about parallelism? given that there are cpus with 
hyper-threading and a physical core may be seen as more than one from a 
parallelism perspective? (ie a singlethreaded MR task would consume at most 1/2 
of a core)

 Define a virtual core unambigiously
 ---

 Key: YARN-1024
 URL: https://issues.apache.org/jira/browse/YARN-1024
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Arun C Murthy

 We need to clearly define the meaning of a virtual core unambiguously so that 
 it's easy to migrate applications between clusters.
 For e.g. here is Amazon EC2 definition of ECU: 
 http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it
 Essentially we need to clearly define a YARN Virtual Core (YVC).
 Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the 
 equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.*

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1055) Handle app recovery differently for AM failures and RM restart

[
https://issues.apache.org/jira/browse/YARN-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741165#comment-13741165
]

Bikas Saha commented on YARN-1055:
--

First of all, like folks have already agreed. This is fundamentally an Oozie
problem. I dont want to add an option to YARN that does not make sense for
YARN by itself. If YARN needs to hack a workaround to fix an Oozie problem, I
would also like to see what Oozie is doing on its part of the bargain. What is
the Oozie jira that fixes this fundamental problem with Oozie?

With the correct settings, this may be a problem only on rare occasions for an
Oozie workflow when an action-am node crashes. IMO its an ok compromise for the
short term while YARN is still not GA.

This issue exists since YARN started and since we started working on RM
restart. If it hasnt been a catastrophic issue till now then IMO it can wait
for some more time till we complete YARN-556. RM restart is work in active
progress and I dont understand why we need to hack an API together when we are
already tracking a proper solution in YARN-556. YARN and Hadoop-1 are different
enough that 1-1 regression matching may not always make sense. Even when it
does, it will be a regression only when YARN goes GA. Until then all of this is
work in progress and users need to be aware of limitations that are known and
being fixed. The cornerstone of the beta release that we all have worked so
hard for is making a viable and stable API that we want to support. Adding a
short term API would go against the basic premise of the beta release.

Any workaround stop gap etc requires code change and maintenance of that code
for future code changes. The request here is for an additional API in
AppSubmissionContext that helps Oozie work around its lack of book-keeping.
Once YARN goes out with beta then this API will have to be maintained forever
since removing an API is backwards incompatible. Given that we are already
committed to fixing this via YARN-556, adding a short term API that will need
to be maintained forever is a disaster and I dont see enough value being added
to suffer through it. We are better off not spending more time on this and
devoting that energy on things like YARN-556 that make real improvements for
everyone.

I really hope this clarifies my position and assures you that we are committed
solving the problem in the correct manner.

Handle app recovery differently for AM failures and RM restart
--

Key: YARN-1055
URL: https://issues.apache.org/jira/browse/YARN-1055
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Affects Versions: 2.1.0-beta
Reporter: Karthik Kambatla

Ideally, we would like to tolerate container, AM, RM failures. App recovery
for AM and RM currently relies on the max-attempts config; tolerating AM
failures requires it to be 1 and tolerating RM failure/restart requires it
to be = 1.
We should handle these two differently, with two separate configs.

[jira] [Commented] (YARN-810) Support CGroup ceiling enforcement on CPU

2013-08-15 Thread Robert Joseph Evans (JIRA)

[
https://issues.apache.org/jira/browse/YARN-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741170#comment-13741170
]

Robert Joseph Evans commented on YARN-810:
--

Sorry I am a bit late to this discussion. I don't like the config to be
global. I think it needs to be on a per container basis.

{quote}There are certain cases where this is desirable. There are also certain
cases where it might be desirable to have a hard limit on CPU usage, and not
allow the process to go above the specified resource requirement, even if it's
available.{quote}

The question is are there ever two different applications running on the same
cluster where it is desirable for one, and not for the other. I believe that
is true. I argued this in YARN-102 where you want to measure how long an
application will take to run under a specific CPU resource request. If I allow
it to go over I will never know how long it would take worst case, and so I
will never know if my config is correct unless I can artificially limit it.
But in production I don't want to run worst case every time, and I don't want a
special test cluster to see what the worst case is.

Support CGroup ceiling enforcement on CPU
-

Key: YARN-810
URL: https://issues.apache.org/jira/browse/YARN-810
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.1.0-beta, 2.0.5-alpha
Reporter: Chris Riccomini
Assignee: Sandy Ryza

Problem statement:
YARN currently lets you define an NM's pcore count, and a pcore:vcore ratio.
Containers are then allowed to request vcores between the minimum and maximum
defined in the yarn-site.xml.
In the case where a single-threaded container requests 1 vcore, with a
pcore:vcore ratio of 1:4, the container is still allowed to use up to 100% of
the core it's using, provided that no other container is also using it. This
happens, even though the only guarantee that YARN/CGroups is making is that
the container will get at least 1/4th of the core.
If a second container then comes along, the second container can take
resources from the first, provided that the first container is still getting
at least its fair share (1/4th).
There are certain cases where this is desirable. There are also certain cases
where it might be desirable to have a hard limit on CPU usage, and not allow
the process to go above the specified resource requirement, even if it's
available.
Here's an RFC that describes the problem in more detail:
http://lwn.net/Articles/336127/
Solution:
As it happens, when CFS is used in combination with CGroups, you can enforce
a ceiling using two files in cgroups:
{noformat}
cpu.cfs_quota_us
cpu.cfs_period_us
{noformat}
The usage of these two files is documented in more detail here:
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-cpu.html
Testing:
I have tested YARN CGroups using the 2.0.5-alpha implementation. By default,
it behaves as described above (it is a soft cap, and allows containers to use
more than they asked for). I then tested CFS CPU quotas manually with YARN.
First, you can see that CFS is in use in the CGroup, based on the file names:
{noformat}
[criccomi@eat1-qa464 ~]$ sudo -u app ls -l /cgroup/cpu/hadoop-yarn/
total 0
-r--r--r-- 1 app app 0 Jun 13 16:46 cgroup.procs
drwxr-xr-x 2 app app 0 Jun 13 17:08 container_1371141151815_0004_01_02
-rw-r--r-- 1 app app 0 Jun 13 16:46 cpu.cfs_period_us
-rw-r--r-- 1 app app 0 Jun 13 16:46 cpu.cfs_quota_us
-rw-r--r-- 1 app app 0 Jun 13 16:46 cpu.rt_period_us
-rw-r--r-- 1 app app 0 Jun 13 16:46 cpu.rt_runtime_us
-rw-r--r-- 1 app app 0 Jun 13 16:46 cpu.shares
-r--r--r-- 1 app app 0 Jun 13 16:46 cpu.stat
-rw-r--r-- 1 app app 0 Jun 13 16:46 notify_on_release
-rw-r--r-- 1 app app 0 Jun 13 16:46 tasks
[criccomi@eat1-qa464 ~]$ sudo -u app cat
/cgroup/cpu/hadoop-yarn/cpu.cfs_period_us
10
[criccomi@eat1-qa464 ~]$ sudo -u app cat
/cgroup/cpu/hadoop-yarn/cpu.cfs_quota_us
-1
{noformat}
Oddly, it appears that the cfs_period_us is set to .1s, not 1s.
We can place processes in hard limits. I have process 4370 running YARN
container container_1371141151815_0003_01_03 on a host. By default, it's
running at ~300% cpu usage.
{noformat}
CPU
4370 criccomi 20 0 1157m 551m 14m S 240.3 0.8 87:10.91 ...
{noformat}
When I set the CFS quote:
{noformat}
echo 1000
/cgroup/cpu/hadoop-yarn/container_1371141151815_0003_01_03/cpu.cfs_quota_us
CPU
4370 criccomi 20 0 1157m 563m 14m S 1.0 0.8 90:08.39

[jira] [Commented] (YARN-1024) Define a virtual core unambigiously

2013-08-15 Thread Sandy Ryza (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741191#comment-13741191
]

Sandy Ryza commented on YARN-1024:
--

bq. To paraphrase your proposal...
Thanks for summarizing. That captures it perfectly.

bq, The one issue that we need to think about is that we'll need to enhance the
schedulers to track how much YCUs are available on which core on any given
node... you could have 5 YCUs in a node but split 3-2-1 across 3 cores. Any
good ideas on how to get to this?
Correct me if I'm wrong, but you're talking about availability based on usage,
not heterogeneous cores within a node, right? If my assumptions about how we
can view CPUs are sufficient, I'm thinking we shouldn't need this, at least for
a start. I.e. if we have two threads to run and two cores to run them on, we
can be agnostic to whether the OS scheduler is running each on its own core or
splitting both across two. The CGroups properties discussed in YARN-810 allow
you to limit the total processing power that a process gets without pinning its
threads to cores. Assigning tasks to cores might matter for things like cache
performance, so I agree it's a useful thing to work on eventually. But I think
any solution will either end up with a decent amount of fragmentation or
require doing some NP-hard combinatorial optimization repeatedly.

Define a virtual core unambigiously
---

Key: YARN-1024
URL: https://issues.apache.org/jira/browse/YARN-1024
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Arun C Murthy

[jira] [Updated] (YARN-1044) used/min/max resources do not display info in the scheduler page


 [ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-1044:
--

Attachment: (was: yarn-1044.patch)

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1044) used/min/max resources do not display info in the scheduler page


 [ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-1044:
--

Attachment: yarn-1044.patch

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1024) Define a virtual core unambigiously

2013-08-15 Thread Arun C Murthy (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741213#comment-13741213
]

Arun C Murthy commented on YARN-1024:
-

bq. Correct me if I'm wrong, but you're talking about availability based on
usage, not heterogeneous cores within a node, right? If my assumptions about
how we can view CPUs are sufficient, I'm thinking we shouldn't need this, at
least for a start.

Ah, good point. Although I would like to really think through implications of
heterogenous nodes in the cluster. In spite, I think there isn't anything here
we'd be blocked on. Anyone disagrees?

Now, the important question.

If we agree, in a broad sense, on [~sandyr]'s proposal - are we happy with our
current APIs, particularly in light of 2.1.0-beta?

One option is for us to use the current (get,set)VirtualCores as the basis for
'cores' or 'parallelism' going fwd and introduce a new (get,set)YCUPerCore?

Is that ok? What do you guys think? Thanks.

Define a virtual core unambigiously
---

Key: YARN-1024
URL: https://issues.apache.org/jira/browse/YARN-1024
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Arun C Murthy

[jira] [Created] (YARN-1066) Improve toString implementation for PBImpls for AHS

2013-08-15 Thread Zhijie Shen (JIRA)

Zhijie Shen created YARN-1066:
-

 Summary: Improve toString implementation for PBImpls for AHS
 Key: YARN-1066
 URL: https://issues.apache.org/jira/browse/YARN-1066
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen


YARN-1045 improves toString implementation for PBImpls, AHS's PBImpls should be 
changed accordingly

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741253#comment-13741253
 ] 

Hadoop QA commented on YARN-1044:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12598242/yarn-1044.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1725//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1725//console

This message is automatically generated.

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741286#comment-13741286
 ] 

Sangjin Lee commented on YARN-1044:
---

I didn't add a new test because it is a small UI change. Let me know if you 
think this would need a new test.

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1024) Define a virtual core unambigiously

2013-08-15 Thread Sandy Ryza (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741289#comment-13741289
 ] 

Sandy Ryza commented on YARN-1024:
--

My opinion is that we shouldn't delay the release for adding in both, and that 
we can add in YCUs in 2.1.1.  If we want to change virtual cores to 'cores' or 
'parallelism', I could post a refactoring patch by EOD.  I also wouldn't cry if 
we left it as virtual cores.

 Define a virtual core unambigiously
 ---

 Key: YARN-1024
 URL: https://issues.apache.org/jira/browse/YARN-1024
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Arun C Murthy

 We need to clearly define the meaning of a virtual core unambiguously so that 
 it's easy to migrate applications between clusters.
 For e.g. here is Amazon EC2 definition of ECU: 
 http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it
 Essentially we need to clearly define a YARN Virtual Core (YVC).
 Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the 
 equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.*

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741292#comment-13741292
 ] 

Omkar Vinit Joshi commented on YARN-1044:
-

Thanks [~sjlee0] I would have liked to see web services related similar issues 
too to be fixed as a part of this ticket as that too is trivial and at least in 
case of capacity Scheduler we probably need to just remove usedResources from 
responseotherwise lgtm +1

{code}
scheduler
schedulerInfo xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; 
xsi:type=capacityScheduler
capacity100.0/capacity
usedCapacity0.0/usedCapacity
maxCapacity100.0/maxCapacity
queueNameroot/queueName
queues
queue xsi:type=capacitySchedulerLeafQueueInfo
capacity100.0/capacity
usedCapacity0.0/usedCapacity
maxCapacity100.0/maxCapacity
absoluteCapacity100.0/absoluteCapacity
absoluteMaxCapacity100.0/absoluteMaxCapacity
absoluteUsedCapacity0.0/absoluteUsedCapacity
numApplications0/numApplications
usedResourcesmemory:0, vCores:0/usedResources
queueNamedefault/queueName
stateRUNNING/state
resourcesUsed
memory0/memory
vCores0/vCores
/resourcesUsed
numActiveApplications0/numActiveApplications
numPendingApplications0/numPendingApplications
numContainers0/numContainers
maxApplications1/maxApplications
maxApplicationsPerUser1/maxApplicationsPerUser
maxActiveApplications7/maxActiveApplications
maxActiveApplicationsPerUser7/maxActiveApplicationsPerUser
userLimit100/userLimit
users/
userLimitFactor1.0/userLimitFactor
/queue
/queues
/schedulerInfo
/scheduler
{code}

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741309#comment-13741309
 ] 

Sangjin Lee commented on YARN-1044:
---

Hmm, do you want to just remove the element from the XML? I thought that there 
is value in showing this field.

I agree that removing it would be trivial, but keeping it and properly escaping 
it seems a little more involved. We would want to escape only if it is an xml 
output (as opposed to json), and that may involve playing with jaxb marshalling.

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741318#comment-13741318
 ] 

Omkar Vinit Joshi commented on YARN-1044:
-

Someone has already fixed it looks like but forgot to remove older...check 
usedResourcesmemory:0, vCores:0/usedResources and resourcesUsed
memory0/memory
vCores0/vCores
/resourcesUsed .. this is trivial can be fixed here itself.. I think we can 
remove usedResources

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741327#comment-13741327
 ] 

Sangjin Lee commented on YARN-1044:
---

Oh I see. The memory and vcores are there instead? I didn't see that. Let me 
see if I can do add that change too. Give me a moment.

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1045) Improve toString implementation for PBImpls

2013-08-15 Thread Siddharth Seth (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated YARN-1045:
-

Fix Version/s: (was: 2.1.1-beta)
   2.1.0-beta

Committed to branch-2.1.0-beta.

 Improve toString implementation for PBImpls
 ---

 Key: YARN-1045
 URL: https://issues.apache.org/jira/browse/YARN-1045
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Jian He
 Fix For: 2.1.0-beta

 Attachments: YARN-1045.1.patch, YARN-1045.patch


 The generic toString implementation that is used in most of the PBImpls 
 {code}getProto().toString().replaceAll(\\n, , ).replaceAll(\\s+,  
 );{code} is rather inefficient - replacing \n and \s to generate a one 
 line string. Instead, we can use 
 {code}TextFormat.shortDebugString(getProto());{code}.
 If we can get this into 2.1.0 - great, otherwise the next release.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1045) Improve toString implementation for PBImpls


[ 
https://issues.apache.org/jira/browse/YARN-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741348#comment-13741348
 ] 

Hudson commented on YARN-1045:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #4271 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/4271/])
Update CHANGES.txt to move YARN-1045 and MAPREDUCE-5352 to the correct version. 
(sseth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514432)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt


 Improve toString implementation for PBImpls
 ---

 Key: YARN-1045
 URL: https://issues.apache.org/jira/browse/YARN-1045
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Jian He
 Fix For: 2.1.0-beta

 Attachments: YARN-1045.1.patch, YARN-1045.patch


 The generic toString implementation that is used in most of the PBImpls 
 {code}getProto().toString().replaceAll(\\n, , ).replaceAll(\\s+,  
 );{code} is rather inefficient - replacing \n and \s to generate a one 
 line string. Instead, we can use 
 {code}TextFormat.shortDebugString(getProto());{code}.
 If we can get this into 2.1.0 - great, otherwise the next release.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-292) ResourceManager throws ArrayIndexOutOfBoundsException while handling CONTAINER_ALLOCATED for application attempt

2013-08-15 Thread Zhijie Shen (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741380#comment-13741380
 ] 

Zhijie Shen commented on YARN-292:
--

bq. FIFO Scheduler uses TreeMap to keep applications in FIFO 
order,ConcurrentHashMap will break this featrue. Right?

AFAIK, FIFO is not controlled by FifoScheduler.applications, and TreeMap cannot 
be used for ordering. Instead, FifoPolicy has a FifoComparator, which can be 
used to sort Schedulable object collection. [~vinodkv], would you please 
confirm it?

 ResourceManager throws ArrayIndexOutOfBoundsException while handling 
 CONTAINER_ALLOCATED for application attempt
 

 Key: YARN-292
 URL: https://issues.apache.org/jira/browse/YARN-292
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Affects Versions: 2.0.1-alpha
Reporter: Devaraj K
Assignee: Zhijie Shen
 Attachments: YARN-292.1.patch, YARN-292.2.patch


 {code:xml}
 2012-12-26 08:41:15,030 ERROR 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler: 
 Calling allocate on removed or non existant application 
 appattempt_1356385141279_49525_01
 2012-12-26 08:41:15,031 ERROR 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type CONTAINER_ALLOCATED for applicationAttempt 
 application_1356385141279_49525
 java.lang.ArrayIndexOutOfBoundsException: 0
   at java.util.Arrays$ArrayList.get(Arrays.java:3381)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AMContainerAllocatedTransition.transition(RMAppAttemptImpl.java:655)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AMContainerAllocatedTransition.transition(RMAppAttemptImpl.java:644)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:490)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:80)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:433)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:414)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
   at java.lang.Thread.run(Thread.java:662)
  {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-292) ResourceManager throws ArrayIndexOutOfBoundsException while handling CONTAINER_ALLOCATED for application attempt

2013-08-15 Thread Zhijie Shen (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741391#comment-13741391
 ] 

Zhijie Shen commented on YARN-292:
--

Sorry, just noticed that

{code}
// Try to assign containers to applications in fifo order
for (Map.EntryApplicationAttemptId, FiCaSchedulerApp e : applications
.entrySet()) {
{code}

There's iteration over the map collection. Probably we can use 
ConcurrentSkipListMap, which is thread safe and preserve the order as TreeMap 
does.

 ResourceManager throws ArrayIndexOutOfBoundsException while handling 
 CONTAINER_ALLOCATED for application attempt
 

 Key: YARN-292
 URL: https://issues.apache.org/jira/browse/YARN-292
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Affects Versions: 2.0.1-alpha
Reporter: Devaraj K
Assignee: Zhijie Shen
 Attachments: YARN-292.1.patch, YARN-292.2.patch


 {code:xml}
 2012-12-26 08:41:15,030 ERROR 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler: 
 Calling allocate on removed or non existant application 
 appattempt_1356385141279_49525_01
 2012-12-26 08:41:15,031 ERROR 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type CONTAINER_ALLOCATED for applicationAttempt 
 application_1356385141279_49525
 java.lang.ArrayIndexOutOfBoundsException: 0
   at java.util.Arrays$ArrayList.get(Arrays.java:3381)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AMContainerAllocatedTransition.transition(RMAppAttemptImpl.java:655)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AMContainerAllocatedTransition.transition(RMAppAttemptImpl.java:644)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:490)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:80)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:433)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:414)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
   at java.lang.Thread.run(Thread.java:662)
  {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1065) NM should provide AuxillaryService data to the container


 [ 
https://issues.apache.org/jira/browse/YARN-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated YARN-1065:
-

Assignee: Xuan Gong

 NM should provide AuxillaryService data to the container
 

 Key: YARN-1065
 URL: https://issues.apache.org/jira/browse/YARN-1065
 Project: Hadoop YARN
  Issue Type: Task
Reporter: Bikas Saha
Assignee: Xuan Gong

 Start container returns auxillary service data to the AM but does not provide 
 the same information to the task itself. It could add that information to the 
 container env with key=service_name and value=service_data. This allows the 
 container to start using the service without having to depend on the AM to 
 send the info to it indirectly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1044) used/min/max resources do not display info in the scheduler page


 [ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-1044:
--

Attachment: yarn-1044.patch

Updated the patch to remove the unnecessary usedResources field.

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741450#comment-13741450
 ] 

Omkar Vinit Joshi commented on YARN-1044:
-

Thanks [~sjlee0] .. just a small request.. please number the patches like 
jira-numberdate(mmdd).number ... you can choose your own format but it 
will help reviewer ..lgtm +1

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741460#comment-13741460
 ] 

Sangjin Lee commented on YARN-1044:
---

Oh OK... I've been following the suggestions in this wiki: 
http://wiki.apache.org/hadoop/HowToContribute :)

I'll do that from now on. Is that OK?

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-881) Priority#compareTo method seems to be wrong.


 [ 
https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-881:
-

Attachment: YARN-881.patch

Patch that change the compareTo method to return other.getPriority() - 
this.getPriority()

 Priority#compareTo method seems to be wrong.
 

 Key: YARN-881
 URL: https://issues.apache.org/jira/browse/YARN-881
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-881.patch


 if lower int value means higher priority, shouldn't we return 
 other.getPriority() - this.getPriority()  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741468#comment-13741468
 ] 

Omkar Vinit Joshi commented on YARN-1044:
-

I see.. yeah this is trivial.. I got this comment in my early days and found it 
to be useful.. :)

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741470#comment-13741470
 ] 

Hadoop QA commented on YARN-1044:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12598290/yarn-1044.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

  
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1726//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1726//console

This message is automatically generated.

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741474#comment-13741474
 ] 

Sangjin Lee commented on YARN-1044:
---

It looks like some tests are broken. Let me look at it and update the patch.

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044.patch, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-897) CapacityScheduler wrongly sorted queues

2013-08-15 Thread Mit Desai (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mit Desai updated YARN-897:
---

Attachment: YARN-897-08152013-br-0.23.patch

Posted patch for branch-0.23

 CapacityScheduler wrongly sorted queues
 ---

 Key: YARN-897
 URL: https://issues.apache.org/jira/browse/YARN-897
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacityscheduler
Affects Versions: 2.0.4-alpha
Reporter: Djellel Eddine Difallah
Assignee: Djellel Eddine Difallah
Priority: Blocker
 Fix For: 2.1.0-beta

 Attachments: TestBugParentQueue.java, 
 YARN-897-08152013-br-0.23.patch, YARN-897-1.patch, YARN-897-2.patch, 
 YARN-897-3.patch, YARN-897-4.patch


 The childQueues of a ParentQueue are stored in a TreeSet where UsedCapacity 
 defines the sort order. This ensures the queue with least UsedCapacity to 
 receive resources next. On containerAssignment we correctly update the order, 
 but we miss to do so on container completions. This corrupts the TreeSet 
 structure, and under-capacity queues might starve for resources.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.


[ 
https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741509#comment-13741509
 ] 

Bikas Saha commented on YARN-881:
-

Fairly straight-foward change. Looks good to me. Jian, did you get a chance to 
run all tests with the patch to make sure we arent regressing?

 Priority#compareTo method seems to be wrong.
 

 Key: YARN-881
 URL: https://issues.apache.org/jira/browse/YARN-881
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-881.patch


 if lower int value means higher priority, shouldn't we return 
 other.getPriority() - this.getPriority()  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.


[ 
https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741511#comment-13741511
 ] 

Jian He commented on YARN-881:
--

All YARN tests passed locally , still running MR tests

 Priority#compareTo method seems to be wrong.
 

 Key: YARN-881
 URL: https://issues.apache.org/jira/browse/YARN-881
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-881.patch


 if lower int value means higher priority, shouldn't we return 
 other.getPriority() - this.getPriority()  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.


[ 
https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741517#comment-13741517
 ] 

Hadoop QA commented on YARN-881:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12598291/YARN-881.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1727//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1727//console

This message is automatically generated.

 Priority#compareTo method seems to be wrong.
 

 Key: YARN-881
 URL: https://issues.apache.org/jira/browse/YARN-881
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-881.patch


 if lower int value means higher priority, shouldn't we return 
 other.getPriority() - this.getPriority()  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.

2013-08-15 Thread Sandy Ryza (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741515#comment-13741515
 ] 

Sandy Ryza commented on YARN-881:
-

I'm OK with the change.  Can testPriority to be renamed to something more 
descriptive, like testComparePriorities?  And can we keep the indentation the 
same on the changed line in FairSchedulder.java?

 Priority#compareTo method seems to be wrong.
 

 Key: YARN-881
 URL: https://issues.apache.org/jira/browse/YARN-881
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-881.patch


 if lower int value means higher priority, shouldn't we return 
 other.getPriority() - this.getPriority()  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1024) Define a virtual core unambigiously

2013-08-15 Thread Robert Joseph Evans (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741518#comment-13741518
 ] 

Robert Joseph Evans commented on YARN-1024:
---

I am fine with that too.

 Define a virtual core unambigiously
 ---

 Key: YARN-1024
 URL: https://issues.apache.org/jira/browse/YARN-1024
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Arun C Murthy

 We need to clearly define the meaning of a virtual core unambiguously so that 
 it's easy to migrate applications between clusters.
 For e.g. here is Amazon EC2 definition of ECU: 
 http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it
 Essentially we need to clearly define a YARN Virtual Core (YVC).
 Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the 
 equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.*

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1044) used/min/max resources do not display info in the scheduler page


 [ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated YARN-1044:
--

Attachment: yarn-1044-20130815.3.patch

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044-20130815.3.patch, 
 yarn-1044.patch, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1044) used/min/max resources do not display info in the scheduler page


[ 
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741602#comment-13741602
 ] 

Hadoop QA commented on YARN-1044:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12598304/yarn-1044-20130815.3.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1729//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1729//console

This message is automatically generated.

 used/min/max resources do not display info in the scheduler page
 

 Key: YARN-1044
 URL: https://issues.apache.org/jira/browse/YARN-1044
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.0.5-alpha
Reporter: Sangjin Lee
Priority: Minor
  Labels: newbie
 Attachments: screenshot.png, yarn-1044-20130815.3.patch, 
 yarn-1044.patch, yarn-1044.patch


 Go to the scheduler page in RM, and click any queue to display the detailed 
 info. You'll find that none of the resources entries (used, min, or max) 
 would display values.
 It is because the values contain brackets ( and ) and are not properly 
 html-escaped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-881) Priority#compareTo method seems to be wrong.


 [ 
https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-881:
-

Attachment: YARN-881.1.patch

uploaded a new patch, fixed Sandy's comments

 Priority#compareTo method seems to be wrong.
 

 Key: YARN-881
 URL: https://issues.apache.org/jira/browse/YARN-881
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-881.1.patch, YARN-881.patch


 if lower int value means higher priority, shouldn't we return 
 other.getPriority() - this.getPriority()  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (YARN-1067) AMRMClient heartbeat interval should not be static

2013-08-15 Thread Siddharth Seth (JIRA)

Siddharth Seth created YARN-1067:


 Summary: AMRMClient heartbeat interval should not be static
 Key: YARN-1067
 URL: https://issues.apache.org/jira/browse/YARN-1067
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.0.5-alpha
Reporter: Siddharth Seth


The heartbeat interval can be modified dynamically - more often when there are 
pending requests, and toned down when the heartbeat is solving no purpose other 
than a ping.
There's a couple of jiras which are trying to change the scheduling loop - at 
which point this becomes useful.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-291) Dynamic resource configuration

2013-08-15 Thread Luke Lu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741713#comment-13741713
 ] 

Luke Lu commented on YARN-291:
--

It'll better if the node resource config is grouped into a repeated field, so 
that you can config multiple nodes in one RPC, which is the common case for 
cluster management systems.

 Dynamic resource configuration
 --

 Key: YARN-291
 URL: https://issues.apache.org/jira/browse/YARN-291
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: nodemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
  Labels: features
 Attachments: Elastic Resources for YARN-v0.2.pdf, 
 YARN-291-AddClientRMProtocolToSetNodeResource-03.patch, 
 YARN-291-all-v1.patch, YARN-291-CoreAndAdmin.patch, 
 YARN-291-core-HeartBeatAndScheduler-01.patch, 
 YARN-291-JMXInterfaceOnNM-02.patch, 
 YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, 
 YARN-291-YARNClientCommandline-04.patch


 The current Hadoop YARN resource management logic assumes per node resource 
 is static during the lifetime of the NM process. Allowing run-time 
 configuration on per node resource will give us finer granularity of resource 
 elasticity. This allows Hadoop workloads to coexist with other workloads on 
 the same hardware efficiently, whether or not the environment is virtualized. 
 More background and design details can be found in attached proposal.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1063) Winutils needs ability to create task as domain user

2013-08-15 Thread Kyle Leckie (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741717#comment-13741717
]

Kyle Leckie commented on YARN-1063:
---

Hi Bikas,
I have updated the Description of this JIRA to include more information. Please
let me know if you have any questions/comments.

Winutils needs ability to create task as domain user

Key: YARN-1063
URL: https://issues.apache.org/jira/browse/YARN-1063
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: trunk-win
Environment: Windows
Reporter: Kyle Leckie
Labels: security
Fix For: trunk-win

Attachments: YARN-732.patch

h1. Summary:
Securing a Hadoop cluster requires constructing some form of security
boundary around the processes executed in YARN containers. Isolation based on
Windows user isolation seems most feasible. This approach is similar to the
approach taken by the existing LinuxContainerExecutor. The current patch to
winutils.exe adds the ability to create a process as a domain user.
h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It
is possible to see this in action by examining the security tone of a
sandboxed process launch be a web browser. Typically the launched process
will have a fully restricted token and need to access machine resources
through a dedicated broker process that enforces a custom security policy.
This broker process mechanism would break compatibility with the typical
Hadoop container process. The Container process must be able to utilize
standard function calls for disk and network IO. I performed some work
looking at ways to ACL the local files to the specific launched without
granting rights to other processes launched on the same machine but found
this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of
WinRT based executables. This method was ruled out due to the lack of
official support for standard windows APIs. At some point in the future
windows may support functionality similar to BSD jails or Linux containers,
at that point support for containers should be added.
h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access
to the user information. For this join the account to the predefined group
labeled Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME
* The launched process will not have rights to the desktop so will not be
able to display any information or create UI.
* The launched process will have no network credentials. Any access of
network resources that requires domain authentication will fail.
h2. Implementation:
Winutils performs the following steps:
# Enable the required privileges for the current process.
# Register as a trusted process with the Local Security Authority (LSA).
# Create a new logon for the user passed on the command line.
# Load/Create a profile on the local machine for the new logon.
# Create a new environment for the new logon.
# Launch the new process in a job with the task name specified and using the
created logon.
# Wait for the JOB to exit.
h2. Future work:
The following work was scoped out of this check in:
* Support for non-domain users or machine that are not domain joined.
* Support for privilege isolation by running the task launcher in a high
privilege service with access over an ACLed named pipe.

[jira] [Updated] (YARN-1063) Winutils needs ability to create task as domain user

2013-08-15 Thread Kyle Leckie (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kyle Leckie updated YARN-1063:
--

Description:
h1. Summary:
Securing a Hadoop cluster requires constructing some form of security boundary
around the processes executed in YARN containers. Isolation based on Windows
user isolation seems most feasible. This approach is similar to the approach
taken by the existing LinuxContainerExecutor. The current patch to winutils.exe
adds the ability to create a process as a domain user.

h1. Alternative Methods considered:
h2. Process rights limited by security token restriction:
On Windows access decisions are made by examining the security token of a
process. It is possible to spawn a process with a restricted security token.
Any of the rights granted by SIDs of the default token may be restricted. It is
possible to see this in action by examining the security tone of a sandboxed
process launch be a web browser. Typically the launched process will have a
fully restricted token and need to access machine resources through a dedicated
broker process that enforces a custom security policy. This broker process
mechanism would break compatibility with the typical Hadoop container process.
The Container process must be able to utilize standard function calls for disk
and network IO. I performed some work looking at ways to ACL the local files to
the specific launched without granting rights to other processes launched on
the same machine but found this to be an overly complex solution.
h2. Relying on APP containers:
Recent versions of windows have the ability to launch processes within an
isolated container. Application containers are supported for execution of WinRT
based executables. This method was ruled out due to the lack of official
support for standard windows APIs. At some point in the future windows may
support functionality similar to BSD jails or Linux containers, at that point
support for containers should be added.

h1. Create As User Feature Description:
h2. Usage:
A new sub command was added to the set of task commands. Here is the syntax:
winutils task createAsUser [TASKNAME] [USERNAME] [COMMAND_LINE]
Some notes:
* The username specified is in the format of user@domain
* The machine executing this command must be joined to the domain of the user
specified
* The domain controller must allow the account executing the command access to
the user information. For this join the account to the predefined group labeled
Pre-Windows 2000 Compatible Access
* The account running the command must have several rights on the local
machine. These can be managed manually using secpol.msc:
** Act as part of the operating system - SE_TCB_NAME
** Replace a process-level token - SE_ASSIGNPRIMARYTOKEN_NAME
** Adjust memory quotas for a process - SE_INCREASE_QUOTA_NAME
* The launched process will not have rights to the desktop so will not be able
to display any information or create UI.
* The launched process will have no network credentials. Any access of network
resources that requires domain authentication will fail.

h2. Implementation:
Winutils performs the following steps:
# Enable the required privileges for the current process.
# Register as a trusted process with the Local Security Authority (LSA).
# Create a new logon for the user passed on the command line.
# Load/Create a profile on the local machine for the new logon.
# Create a new environment for the new logon.
# Launch the new process in a job with the task name specified and using the
created logon.
# Wait for the JOB to exit.

h2. Future work:
The following work was scoped out of this check in:
* Support for non-domain users or machine that are not domain joined.
* Support for privilege isolation by running the task launcher in a high
privilege service with access over an ACLed named pipe.

was:Task isolation requires the ability to launch tasks in the context of a
particular domain user.

Winutils needs ability to create task as domain user

Attachments: YARN-732.patch

[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.


[ 
https://issues.apache.org/jira/browse/YARN-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741766#comment-13741766
 ] 

Hadoop QA commented on YARN-881:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12598323/YARN-881.1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1730//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1730//console

This message is automatically generated.

 Priority#compareTo method seems to be wrong.
 

 Key: YARN-881
 URL: https://issues.apache.org/jira/browse/YARN-881
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-881.1.patch, YARN-881.patch


 if lower int value means higher priority, shouldn't we return 
 other.getPriority() - this.getPriority()  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-881) Priority#compareTo method seems to be wrong.