date:20120713


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413518#comment-13413518
 ] 

Arun C Murthy commented on MAPREDUCE-4334:
--

bq. Clearly, we need to support taskset for platforms on which cgroups isn't 
supported e.g. RHEL5. For taskset you need super-user privs - would you prefer 
packages to do it too?

I meant to say: for taskset we clearly need to go via LCE at runtime.

 Add support for CPU isolation/monitoring of containers
 --

 Key: MAPREDUCE-4334
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Andrew Ferguson
 Attachments: MAPREDUCE-4334-pre1.patch, 
 MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, 
 MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch


 Once we get in MAPREDUCE-4327, it will be important to actually enforce 
 limits on CPU consumption of containers. 
 Several options spring to mind:
 # taskset (RHEL5+)
 # cgroups (RHEL6+)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's that are not managed by it


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated MAPREDUCE-4427:
--

Status: Patch Available  (was: Open)

 Enable the RM to work with AM's that are not managed by it
 --

 Key: MAPREDUCE-4427
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
  Labels: mrv2
 Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch


 Currently, the RM itself manages the AM by allocating a container for it and 
 negotiating the launch on the NodeManager and manages the AM lifecycle. 
 Thereafter, the AM negotiates resources with the RM and launches tasks to do 
 the real work.
 It would be a useful improvement to enhance this model by allowing the AM to 
 be launched independently by the client without requiring the RM. These AM's 
 would be launched on a gateway machine that can talk to the cluster. This 
 would open up new use cases such as the following
 1) Easy debugging of AM, specially during initial development. Having the AM 
 launched on an arbitrary cluster node makes it hard to looks at logs or 
 attach a debugger to the AM. If it can be launched locally then these tasks 
 would be easier.
 2) Running AM's that need special privileges that may not be available on 
 machines managed by the NodeManager

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's that are not managed by it


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated MAPREDUCE-4427:
--

Status: Open  (was: Patch Available)

 Enable the RM to work with AM's that are not managed by it
 --

 Key: MAPREDUCE-4427
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
  Labels: mrv2
 Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch


 Currently, the RM itself manages the AM by allocating a container for it and 
 negotiating the launch on the NodeManager and manages the AM lifecycle. 
 Thereafter, the AM negotiates resources with the RM and launches tasks to do 
 the real work.
 It would be a useful improvement to enhance this model by allowing the AM to 
 be launched independently by the client without requiring the RM. These AM's 
 would be launched on a gateway machine that can talk to the cluster. This 
 would open up new use cases such as the following
 1) Easy debugging of AM, specially during initial development. Having the AM 
 launched on an arbitrary cluster node makes it hard to looks at logs or 
 attach a debugger to the AM. If it can be launched locally then these tasks 
 would be easier.
 2) Running AM's that need special privileges that may not be available on 
 machines managed by the NodeManager

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's that are not managed by it


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated MAPREDUCE-4427:
--

Attachment: MAPREDUCE-4427-2.patch

Attaching patch with some clarifying comments removed for final submission.


 Enable the RM to work with AM's that are not managed by it
 --

 Key: MAPREDUCE-4427
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
  Labels: mrv2
 Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch


 Currently, the RM itself manages the AM by allocating a container for it and 
 negotiating the launch on the NodeManager and manages the AM lifecycle. 
 Thereafter, the AM negotiates resources with the RM and launches tasks to do 
 the real work.
 It would be a useful improvement to enhance this model by allowing the AM to 
 be launched independently by the client without requiring the RM. These AM's 
 would be launched on a gateway machine that can talk to the cluster. This 
 would open up new use cases such as the following
 1) Easy debugging of AM, specially during initial development. Having the AM 
 launched on an arbitrary cluster node makes it hard to looks at logs or 
 attach a debugger to the AM. If it can be launched locally then these tasks 
 would be easier.
 2) Running AM's that need special privileges that may not be available on 
 machines managed by the NodeManager

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's that are not managed by it

[
https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413525#comment-13413525
]

Arun C Murthy commented on MAPREDUCE-4427:
--

+1, lgtm! Thanks for the patch, this is very cool!

Enable the RM to work with AM's that are not managed by it
--

Key: MAPREDUCE-4427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
Labels: mrv2
Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch

Currently, the RM itself manages the AM by allocating a container for it and
negotiating the launch on the NodeManager and manages the AM lifecycle.
Thereafter, the AM negotiates resources with the RM and launches tasks to do
the real work.
It would be a useful improvement to enhance this model by allowing the AM to
be launched independently by the client without requiring the RM. These AM's
would be launched on a gateway machine that can talk to the cluster. This
would open up new use cases such as the following
1) Easy debugging of AM, specially during initial development. Having the AM
launched on an arbitrary cluster node makes it hard to looks at logs or
attach a debugger to the AM. If it can be launched locally then these tasks
would be easier.
2) Running AM's that need special privileges that may not be available on
machines managed by the NodeManager

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4436) AppRejectedTransition does not unregister app from master service and scheduler

Bikas Saha created MAPREDUCE-4436:
-

 Summary: AppRejectedTransition does not unregister app from master 
service and scheduler
 Key: MAPREDUCE-4436
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4436
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.0-alpha, 0.23.1, 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha


AttemptStartedTransition() adds the app to the ApplicationMasterService and 
scheduler. when the scheduler rejects the app then AppRejectedTransition() 
forgets to unregister it from the ApplicationMasterService.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4309) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413526#comment-13413526
 ] 

Bikas Saha commented on MAPREDUCE-4309:
---

1) AppSchedulingInfo.java
In allocateNodeGroupLocal(...), why not call 
decrementOutstanding(requests.get(priority).get(RMNode.ANY)) instead of the 
following
{code}
// Do not remove ANY
ResourceRequest offSwitchRequest = requests.get(priority).get(
RMNode.ANY);
offSwitchRequest.setNumContainers(offSwitchRequest.getNumContainers() 
{code}

2) Many methods/variables of multilple classes (LeafQueue/FifoScheduler etc) 
have been made protected but I dont see them subclassed anywhere. Whats the 
purpose of those changes?

3) Some objects are not being created via reflection and config. Its not clear 
what the purpose of those are?

 Make locatlity in YARN's container assignment and task scheduling pluggable 
 for other deployment topology
 -

 Key: MAPREDUCE-4309
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4309
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0-alpha
Reporter: Junping Du
Assignee: Junping Du
 Attachments: 
 HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, 
 MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, MAPREDUCE-4309-v4.patch, 
 MAPREDUCE-4309.patch


 There are several classes in YARN’s container assignment and task scheduling 
 algorithms that relate to data locality which were updated to give preference 
 to running a container on other locality besides node-local and rack-local 
 (like nodegroup-local). This propose to make these data structure/algorithms 
 pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class 
 ScheduledRequests was made a package level class to it would be easier to 
 create a subclass, ScheduledRequestsWithNodeGroup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4436) AppRejectedTransition does not unregister app from master service and scheduler


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413528#comment-13413528
 ] 

Arun C Murthy commented on MAPREDUCE-4436:
--

Good catch!

 AppRejectedTransition does not unregister app from master service and 
 scheduler
 ---

 Key: MAPREDUCE-4436
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4436
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.1, 2.0.0-alpha, 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha

 AttemptStartedTransition() adds the app to the ApplicationMasterService and 
 scheduler. when the scheduler rejects the app then AppRejectedTransition() 
 forgets to unregister it from the ApplicationMasterService.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4403) Adding test case for resubmission of jobs in TestRecoveryManager


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413539#comment-13413539
 ] 

Konstantin Shvachko commented on MAPREDUCE-4403:


+1 looks good

 Adding test case for resubmission of jobs in TestRecoveryManager
 

 Key: MAPREDUCE-4403
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4403
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Attachments: MAPREDUCE-4403-22-1.patch, MAPREDUCE-4403-22.patch


 In Hadoop 22 Test recovery Manager does not have resubmission test case which 
 checks after the resubmission jobs get succeeded.
 There is some refactoring is also needed. 
 Thanks,
 Mayank

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-4422:
---

  Resolution: Fixed
   Fix Version/s: 2.0.1-alpha
Target Version/s:   (was: 2.0.1-alpha)
  Status: Resolved  (was: Patch Available)

Committed to branch-2 and trunk. Thanks Ahmed!

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4309) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

2012-07-13 Thread Junping Du (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413546#comment-13413546
 ] 

Junping Du commented on MAPREDUCE-4309:
---

Bikas, thanks for your comments. Please see my reply:
1). That's a good point. I have this code before decrementOutstanding() is 
there. I will fix this in following patch.   
2 and 3). subclass is put in as plug-in patch which is tracked by 
MAPREDUCE-4310. With seeing that patch as well as the proposal in Umbrella jira 
(HADOOP-8468), it will help you understand the work we are doing in this jira.
 

 Make locatlity in YARN's container assignment and task scheduling pluggable 
 for other deployment topology
 -

 Key: MAPREDUCE-4309
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4309
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0-alpha
Reporter: Junping Du
Assignee: Junping Du
 Attachments: 
 HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, 
 MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, MAPREDUCE-4309-v4.patch, 
 MAPREDUCE-4309.patch


 There are several classes in YARN’s container assignment and task scheduling 
 algorithms that relate to data locality which were updated to give preference 
 to running a container on other locality besides node-local and rack-local 
 (like nodegroup-local). This propose to make these data structure/algorithms 
 pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class 
 ScheduledRequests was made a package level class to it would be easier to 
 create a subclass, ScheduledRequestsWithNodeGroup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's that are not managed by it

2012-07-13 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413547#comment-13413547
]

Hadoop QA commented on MAPREDUCE-4427:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12536339/MAPREDUCE-4427-2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 4 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

-1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager

hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2585//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2585//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2585//console

This message is automatically generated.

Enable the RM to work with AM's that are not managed by it
--

[jira] [Updated] (MAPREDUCE-4403) Adding test case for resubmission of jobs in TestRecoveryManager


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko updated MAPREDUCE-4403:
---

Fix Version/s: 0.22.1
 Hadoop Flags: Reviewed

I just committed this to branch 0.22.1. Thank you Mayank.
Do we need this for trunk or other versions?

 Adding test case for resubmission of jobs in TestRecoveryManager
 

 Key: MAPREDUCE-4403
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4403
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Fix For: 0.22.1

 Attachments: MAPREDUCE-4403-22-1.patch, MAPREDUCE-4403-22.patch


 In Hadoop 22 Test recovery Manager does not have resubmission test case which 
 checks after the resubmission jobs get succeeded.
 There is some refactoring is also needed. 
 Thanks,
 Mayank

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413563#comment-13413563
 ] 

Hudson commented on MAPREDUCE-4422:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2527 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2527/])
MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default value 
in YarnConfiguration. Contributed by Ahmed Radwan. (harsh) (Revision 1361061)

 Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361061
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413565#comment-13413565
 ] 

Hudson commented on MAPREDUCE-4422:
---

Integrated in Hadoop-Common-trunk-Commit #2461 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2461/])
MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default value 
in YarnConfiguration. Contributed by Ahmed Radwan. (harsh) (Revision 1361061)

 Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361061
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4404) Adding Test case for TestMRJobClient to verify the user name


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko updated MAPREDUCE-4404:
---

Hadoop Flags: Reviewed

I just committed this to branch 0.22.1. Thank you Mayank.
Is it also targeted for trunk?

 Adding Test case for TestMRJobClient to verify the user name
 

 Key: MAPREDUCE-4404
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4404
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Fix For: 0.22.1

 Attachments: MAPREDUCE-4404-22.patch


 Adding Test case for TestMRJobClient to verify the user name

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun C Murthy reopened MAPREDUCE-4422:
--


Uh, Harsh/Ahmed, I'm confused... the newly added variable 
DEFAULT_YARN_APPLICATION_CLASSPATH isn't being used anywhere? The point of 
adding this was to use it, at least, in MRApps.setMRFrameworkClasspath?

Also, not sure whether this was introduced via MAPREDUCE-3505, but we should 
use Configuration.getStrings rather than do a 'manual' String.split to parse 
this out.

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4405) Adding test case for HierarchicalQueue in TestJobQueueClient


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413572#comment-13413572
 ] 

Konstantin Shvachko commented on MAPREDUCE-4405:


assertNotNull for the resulting queues is good, but you can also verify that 
the total number of queues is as expected.

 Adding test case for HierarchicalQueue in TestJobQueueClient
 

 Key: MAPREDUCE-4405
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4405
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Attachments: MAPREDUCE-4405-22.patch


 Adding test case for HierarchicalQueue in TestJobQueueClient

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413593#comment-13413593
 ] 

Harsh J commented on MAPREDUCE-4422:


/me facepalms at not seeing that. Sorry.

I have reverted MAPREDUCE-4422 from both branch-2 and trunk, for until
your comments are all addressed.

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration

2012-07-13 Thread Ahmed Radwan (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413594#comment-13413594
]

Ahmed Radwan commented on MAPREDUCE-4422:
-

the newly added variable DEFAULT_YARN_APPLICATION_CLASSPATH isn't being used
anywhere? The point of adding this was to use it, at least, in
MRApps.setMRFrameworkClasspath?

MRApps.setMRFrameworkClasspath reads the value from the conf files through
conf.get(YarnConfiguration.YARN_APPLICATION_CLASSPATH), and if not set, the
default value will be picked from yarn-default.xml. Making it reads the
DEFAULT_YARN_APPLICATION_CLASSPATH will defeat the purpose of having it
configurable. Or do you just mean modifying the call to be
conf.get(YarnConfiguration.YARN_APPLICATION_CLASSPATH,
DEFAULT_YARN_APPLICATION_CLASSPATH), which will not make a difference in this
case as the default is already picked if the property is not set.

we should use Configuration.getStrings rather than do a 'manual' String.split
to parse this out.

Configuration.getString uses StringTokenizer to achieve exactly what
String.split will do here. I am not sure if one is more efficient than the
other, but I can change the it if you know it is better. I'll upload a new
patch with the updates.

YARN_APPLICATION_CLASSPATH needs a documented default value in
YarnConfiguration

Key: MAPREDUCE-4422
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
Fix For: 2.0.1-alpha

Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch

MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
However, we didn't add a default value to YarnConfiguration, as-is the norm.
Ran into it while investigating MAPREDUCE-4421.

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413595#comment-13413595
 ] 

Hudson commented on MAPREDUCE-4422:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #2481 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2481/])
MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default value 
in YarnConfiguration. Contributed by Ahmed Radwan. (harsh) (Revision 1361061)

 Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361061
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413601#comment-13413601
 ] 

Hudson commented on MAPREDUCE-4422:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2528 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2528/])
Reverting MAPREDUCE-4422 from trunk until all comments are addressed 
(Revision 1361107)

 Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361107
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413603#comment-13413603
 ] 

Hudson commented on MAPREDUCE-4422:
---

Integrated in Hadoop-Common-trunk-Commit #2462 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2462/])
Reverting MAPREDUCE-4422 from trunk until all comments are addressed 
(Revision 1361107)

 Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361107
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-4422:
---

Attachment: MAPREDUCE-4422_rev3.patch

Here's a patch with Arun's comments specifically addressed.

Arun/Ahmed - Are there other spots and/or tests we should be using this default 
string at?

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch, 
 MAPREDUCE-4422_rev3.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration

2012-07-13 Thread Ahmed Radwan (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-4422:


Attachment: MAPREDUCE-4422_rev3.patch

Thanks Arun and Harsh, Here is the updated patch with Arun comments addressed. 
I have also spotted another use in distributedshell client that could be 
updated and I added that to the patch.

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch, 
 MAPREDUCE-4422_rev3.patch, MAPREDUCE-4422_rev3.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4395) Possible NPE at ClientDistributedCacheManager#determineTimestamps

2012-07-13 Thread Bhallamudi Venkata Siva Kamesh (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-4395:
--

Affects Version/s: (was: 3.0.0)
   trunk
   2.0.0-alpha
   Status: Patch Available  (was: Open)

 Possible NPE at ClientDistributedCacheManager#determineTimestamps
 -

 Key: MAPREDUCE-4395
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4395
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: distributed-cache, job submission, mrv2
Affects Versions: 2.0.0-alpha, trunk
Reporter: Bhallamudi Venkata Siva Kamesh
Assignee: Bhallamudi Venkata Siva Kamesh
Priority: Critical
 Attachments: MAPREDUCE-4395.patch


 {code:title=ClientDistributedCacheManager#determineTimestamps|borderStyle=solid}
 URI[] tfiles = DistributedCache.getCacheFiles(job);
 {code}
 It may be possible that tfiles array contains *null* as it's entry, and 
 subsequently leads to NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4395) Possible NPE at ClientDistributedCacheManager#determineTimestamps

2012-07-13 Thread Bhallamudi Venkata Siva Kamesh (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-4395:
--

Attachment: MAPREDUCE-4395.patch

Submitting patch, please review.

 Possible NPE at ClientDistributedCacheManager#determineTimestamps
 -

 Key: MAPREDUCE-4395
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4395
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: distributed-cache, job submission, mrv2
Affects Versions: 2.0.0-alpha, trunk
Reporter: Bhallamudi Venkata Siva Kamesh
Assignee: Bhallamudi Venkata Siva Kamesh
Priority: Critical
 Attachments: MAPREDUCE-4395.patch


 {code:title=ClientDistributedCacheManager#determineTimestamps|borderStyle=solid}
 URI[] tfiles = DistributedCache.getCacheFiles(job);
 {code}
 It may be possible that tfiles array contains *null* as it's entry, and 
 subsequently leads to NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413622#comment-13413622
 ] 

Hudson commented on MAPREDUCE-4422:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #2482 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2482/])
Reverting MAPREDUCE-4422 from trunk until all comments are addressed 
(Revision 1361107)

 Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361107
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch, 
 MAPREDUCE-4422_rev3.patch, MAPREDUCE-4422_rev3.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4395) Possible NPE at ClientDistributedCacheManager#determineTimestamps

2012-07-13 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413627#comment-13413627
]

Hadoop QA commented on MAPREDUCE-4395:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536357/MAPREDUCE-4395.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests in
hadoop-common-project/hadoop-common:

org.apache.hadoop.io.file.tfile.TestTFileByteArrays

org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2586//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2586//console

This message is automatically generated.

Possible NPE at ClientDistributedCacheManager#determineTimestamps
-

Key: MAPREDUCE-4395
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4395
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, job submission, mrv2
Affects Versions: 2.0.0-alpha, trunk
Reporter: Bhallamudi Venkata Siva Kamesh
Assignee: Bhallamudi Venkata Siva Kamesh
Priority: Critical
Attachments: MAPREDUCE-4395.patch

{code:title=ClientDistributedCacheManager#determineTimestamps|borderStyle=solid}
URI[] tfiles = DistributedCache.getCacheFiles(job);
{code}
It may be possible that tfiles array contains *null* as it's entry, and
subsequently leads to NPE.

[jira] [Commented] (MAPREDUCE-4395) Possible NPE at ClientDistributedCacheManager#determineTimestamps

2012-07-13 Thread Bhallamudi Venkata Siva Kamesh (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413648#comment-13413648
 ] 

Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-4395:
---

test failures are unrelated to this patch.

 Possible NPE at ClientDistributedCacheManager#determineTimestamps
 -

 Key: MAPREDUCE-4395
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4395
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: distributed-cache, job submission, mrv2
Affects Versions: 2.0.0-alpha, trunk
Reporter: Bhallamudi Venkata Siva Kamesh
Assignee: Bhallamudi Venkata Siva Kamesh
Priority: Critical
 Attachments: MAPREDUCE-4395.patch


 {code:title=ClientDistributedCacheManager#determineTimestamps|borderStyle=solid}
 URI[] tfiles = DistributedCache.getCacheFiles(job);
 {code}
 It may be possible that tfiles array contains *null* as it's entry, and 
 subsequently leads to NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413665#comment-13413665
 ] 

Hudson commented on MAPREDUCE-4422:
---

Integrated in Hadoop-Hdfs-trunk #1102 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1102/])
Reverting MAPREDUCE-4422 from trunk until all comments are addressed 
(Revision 1361107)
MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default value in 
YarnConfiguration. Contributed by Ahmed Radwan. (harsh) (Revision 1361061)

 Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361107
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml

harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361061
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch, 
 MAPREDUCE-4422_rev3.patch, MAPREDUCE-4422_rev3.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413670#comment-13413670
 ] 

Hudson commented on MAPREDUCE-3451:
---

Integrated in Hadoop-Hdfs-trunk #1102 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1102/])
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 
1361020)

 Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361020
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
*

[jira] [Commented] (MAPREDUCE-4416) Some tests fail if Clover is enabled


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413671#comment-13413671
 ] 

Hudson commented on MAPREDUCE-4416:
---

Integrated in Hadoop-Hdfs-trunk #1102 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1102/])
MAPREDUCE-4416. Some tests fail if Clover is enabled (Kihwal Lee via bobby) 
(Revision 1360735)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1360735
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestIndexCache.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml
* /hadoop/common/trunk/hadoop-project/pom.xml


 Some tests fail if Clover is enabled
 

 Key: MAPREDUCE-4416
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4416.patch.txt, mapreduce-4416.patch.txt


 There are number of tests running under hadoop-mapreduce-client-jobclient 
 that fail if Clover is enabled. Whenever a job is launched, AM doesn't start 
 because it can't locate the clover jar file.
 I thought MAPREDUCE-4253 had something to do with this, but I can reproduce 
 the issue on an older revision. Although unrelated, MAPREDUCE-4253 does have 
 a problem and it has been reported to the jira.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

2012-07-13 Thread Avner BenHanoch (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413700#comment-13413700
]

Avner BenHanoch commented on MAPREDUCE-4049:

Hi Arun,

Thanks for your comperhensive review. I accept all your comments, and I'll
provide new patch soon.

I'll be happy to submit a corresponding patch for hadoop-2. Is it possible to
start with hadoop-1 patch in order to have it on time for 1.1.0? - I'll be
eager to do the hadoop-2 patch immediately after it.

1. We use taskDone/jobDone as optional messages to the plugin. We have
experimental variant of the plugin that uses these messages for optimizing
cache handling.

2. ok

3. I'll include mapred-default.xml in the patch. I'll use default values for
the the new config.

Thank you for working with me on it,
Avner

plugin for generic shuffle service
--

Key: MAPREDUCE-4049
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
Labels: merge, plugin, rdma, shuffle
Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch,
HADOOP-1.1.patch, HADOOP-1.x.y-review-oriented.patch, Hadoop Shuffle Consumer
Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml

Support generic shuffle service as set of two plugins: ShuffleProvider
ShuffleConsumer.
This will satisfy the following needs:
# Better shuffle and merge performance. For example: we are working on
shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE,
or Infiniband) instead of using the current HTTP shuffle. Based on the fast
RDMA shuffle, the plugin can also utilize a suitable merge approach during
the intermediate merges. Hence, getting much better performance.
# Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden
dependency of NodeManager with a specific version of mapreduce shuffle
(currently targeted to 0.24.0).
References:
# Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu
from Auburn University with others,
[http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
# I am attaching 2 documents with suggested Top Level Design for both plugins
(currently, based on 1.0 branch)

[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's that are not managed by it

2012-07-13 Thread Thomas Graves (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413716#comment-13413716
 ] 

Thomas Graves commented on MAPREDUCE-4427:
--

I only skimmed the patch to see how to use this but don't see any 
documentation?'

Is there a way to make sure folks can't use this if admins chose?  

Is there another jira to actually add this to an existing app master?

 Enable the RM to work with AM's that are not managed by it
 --

 Key: MAPREDUCE-4427
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
  Labels: mrv2
 Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch


 Currently, the RM itself manages the AM by allocating a container for it and 
 negotiating the launch on the NodeManager and manages the AM lifecycle. 
 Thereafter, the AM negotiates resources with the RM and launches tasks to do 
 the real work.
 It would be a useful improvement to enhance this model by allowing the AM to 
 be launched independently by the client without requiring the RM. These AM's 
 would be launched on a gateway machine that can talk to the cluster. This 
 would open up new use cases such as the following
 1) Easy debugging of AM, specially during initial development. Having the AM 
 launched on an arbitrary cluster node makes it hard to looks at logs or 
 attach a debugger to the AM. If it can be launched locally then these tasks 
 would be easier.
 2) Running AM's that need special privileges that may not be available on 
 machines managed by the NodeManager

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4403) Adding test case for resubmission of jobs in TestRecoveryManager


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413743#comment-13413743
 ] 

Hudson commented on MAPREDUCE-4403:
---

Integrated in Hadoop-Mapreduce-22-branch #109 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/109/])
MAPREDUCE-4403. Test case for job resubmission in TestRecoveryManager. 
Contributed by Mayank Bansal. (Revision 1361069)

 Result = SUCCESS
shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361069
Files : 
* /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
* 
/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestRecoveryManager.java


 Adding test case for resubmission of jobs in TestRecoveryManager
 

 Key: MAPREDUCE-4403
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4403
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Fix For: 0.22.1

 Attachments: MAPREDUCE-4403-22-1.patch, MAPREDUCE-4403-22.patch


 In Hadoop 22 Test recovery Manager does not have resubmission test case which 
 checks after the resubmission jobs get succeeded.
 There is some refactoring is also needed. 
 Thanks,
 Mayank

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4404) Adding Test case for TestMRJobClient to verify the user name


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413744#comment-13413744
 ] 

Hudson commented on MAPREDUCE-4404:
---

Integrated in Hadoop-Mapreduce-22-branch #109 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/109/])
MAPREDUCE-4404. Test case for TestMRJobClient to verify user name. 
Contributed by Mayank Bansal. (Revision 1361071)

 Result = SUCCESS
shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361071
Files : 
* /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
* 
/hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java


 Adding Test case for TestMRJobClient to verify the user name
 

 Key: MAPREDUCE-4404
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4404
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Fix For: 0.22.1

 Attachments: MAPREDUCE-4404-22.patch


 Adding Test case for TestMRJobClient to verify the user name

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2454) Allow external sorter plugin for MR

2012-07-13 Thread Mariappan Asokan (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413747#comment-13413747
 ] 

Mariappan Asokan commented on MAPREDUCE-2454:
-

Thanks for your comments Arun.  I will start looking at the failing tests.

-- Asokan

 Allow external sorter plugin for MR
 ---

 Key: MAPREDUCE-2454
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Mariappan Asokan
Priority: Minor
  Labels: features, performance, plugin, sort
 Attachments: HadoopSortPlugin.pdf, KeyValueIterator.java, 
 MR-2454-trunkPatchPreview.gz, MapOutputSorter.java, 
 MapOutputSorterAbstract.java, ReduceInputSorter.java, mapreduce-2454.patch, 
 mr-2454-on-mr-279-build82.patch.gz


 Define interfaces and some abstract classes in the Hadoop framework to 
 facilitate external sorter plugins both on the Map and Reduce sides.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration

[
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413751#comment-13413751
]

Arun C Murthy commented on MAPREDUCE-4422:
--

bq. Or do you just mean modifying the call to be
conf.get(YarnConfiguration.YARN_APPLICATION_CLASSPATH,
DEFAULT_YARN_APPLICATION_CLASSPATH),

Ahmed - yes. Apologies if I wasn't clear, that was what I meant.

bq. which will not make a difference in this case as the default is already
picked if the property is not set.

Ahmed - the point of doing this is essentially documentation. We follow the
convention all over the codebase so that developers quickly grok the flow; and
hence the jira itself.

bq. Configuration.getString uses StringTokenizer to achieve exactly what
String.split will do here. I am not sure if one is more efficient than the
other,

Ahmed - the idea behind adding Configuration.getStrings was to re-use existing
apis rather than hand-craft it for modularity/reuse, not efficiency. As you can
see, we use it in a number of places all over the codebase for similar
scenarios to parse comma-delimited values.

YARN_APPLICATION_CLASSPATH needs a documented default value in
YarnConfiguration

Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch,
MAPREDUCE-4422_rev3.patch, MAPREDUCE-4422_rev3.patch

MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
However, we didn't add a default value to YarnConfiguration, as-is the norm.
Ran into it while investigating MAPREDUCE-4421.

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413755#comment-13413755
 ] 

Hudson commented on MAPREDUCE-4422:
---

Integrated in Hadoop-Mapreduce-trunk #1135 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1135/])
Reverting MAPREDUCE-4422 from trunk until all comments are addressed 
(Revision 1361107)
MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default value in 
YarnConfiguration. Contributed by Ahmed Radwan. (harsh) (Revision 1361061)

 Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361107
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml

harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361061
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch, 
 MAPREDUCE-4422_rev3.patch, MAPREDUCE-4422_rev3.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4416) Some tests fail if Clover is enabled


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413761#comment-13413761
 ] 

Hudson commented on MAPREDUCE-4416:
---

Integrated in Hadoop-Mapreduce-trunk #1135 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1135/])
MAPREDUCE-4416. Some tests fail if Clover is enabled (Kihwal Lee via bobby) 
(Revision 1360735)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1360735
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestIndexCache.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml
* /hadoop/common/trunk/hadoop-project/pom.xml


 Some tests fail if Clover is enabled
 

 Key: MAPREDUCE-4416
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4416.patch.txt, mapreduce-4416.patch.txt


 There are number of tests running under hadoop-mapreduce-client-jobclient 
 that fail if Clover is enabled. Whenever a job is launched, AM doesn't start 
 because it can't locate the clover jar file.
 I thought MAPREDUCE-4253 had something to do with this, but I can reproduce 
 the issue on an older revision. Although unrelated, MAPREDUCE-4253 does have 
 a problem and it has been reported to the jira.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413760#comment-13413760
 ] 

Hudson commented on MAPREDUCE-3451:
---

Integrated in Hadoop-Mapreduce-trunk #1135 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1135/])
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 
1361020)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361020
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
*

[jira] [Resolved] (MAPREDUCE-4430) Adding child queues to any queue need the process restart ./yarn rmadmin -refreshQueues throws IO exception


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-4430.
---

Resolution: Duplicate

Duplicate of MAPREDUCE-3410.

 Adding child queues to any queue need the process restart ./yarn rmadmin 
 -refreshQueues throws IO exception
 -

 Key: MAPREDUCE-4430
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4430
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.0-alpha
Reporter: Nishan Shetty

 1.Configure different queues for capacity scheduler say a,b under root.
 2.Start the process
 3.Now add the child queue b1,b2 under b
 4.Now do refresh queues with command ./yarn rmadmin -refreshQueues
 Observed that it throws the following IO exception
 {noformat}
 java.io.IOException: Failed to re-init queues
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:216)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:174)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.api.impl.pb.service.RMAdminProtocolPBServiceImpl.refreshQueues(RMAdminProtocolPBServiceImpl.java:62)
 at 
 org.apache.hadoop.yarn.proto.RMAdminProtocol$RMAdminProtocolService$2.callBlockingMethod(RMAdminProtocol.java:122)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686)
 Caused by: java.io.IOException: Trying to reinitialize root.b from root.b
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.reinitialize(LeafQueue.java:554)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.reinitialize(ParentQueue.java:387)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:257)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:213)
 ... 11 more
  at LocalTrace:
 org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: 
 Failed to re-init queues
 at 
 org.apache.hadoop.yarn.factories.impl.pb.YarnRemoteExceptionFactoryPBImpl.createYarnRemoteException(YarnRemoteExceptionFactoryPBImpl.java:50)
 at 
 org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:40)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:184)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.api.impl.pb.service.RMAdminProtocolPBServiceImpl.refreshQueues(RMAdminProtocolPBServiceImpl.java:62)
 at 
 org.apache.hadoop.yarn.proto.RMAdminProtocol$RMAdminProtocolService$2.callBlockingMethod(RMAdminProtocol.java:122)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686)
 Caused by: 
 org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: Trying 
 to reinitialize root.b from root.b
 at 
 org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.getCause(YarnRemoteExceptionPBImpl.java:94)
 at 
 org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.getCause(YarnRemoteExceptionPBImpl.java:32)
 at java.lang.Throwable.printStackTrace(Throwable.java:514)
 at 
 org.apache.hadoop.yarn.exceptions.YarnRemoteException.printStackTrace(YarnRemoteException.java:48)
 at 
 org.apache.hadoop.util.StringUtils.stringifyException(StringUtils.java:69)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1715)

[jira] [Commented] (MAPREDUCE-3410) Capacity scheduler reconfiguration of queues does not work for add sub-queues to an existing queue


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413768#comment-13413768
 ] 

Jason Lowe commented on MAPREDUCE-3410:
---

Part of the issue here is that adding child queues to what was a leaf queue has 
some similarities with the delete queue case, and the delete queue case 
currently is not supported.  Applications only run in leaf queues, so changing 
a leaf queue into a parent queue is sort of like deleting the leaf queue and 
adding a parent queuethat happens to have the same name.  Like the delete 
scenario, we have to determine what to do with the applications currently 
running on the leaf queue when the reconfig occurs.  Do we terminate the active 
applications?  Stop accepting new applications but let the current ones drain 
before reconfiguring?  In the short term, I think it's a documentation issue to 
state it's not a supported operation.  Once support for queue delete is added, 
I think support for this would be straightforward (or vice-versa).

 Capacity scheduler reconfiguration of queues does not work for add sub-queues 
 to an existing queue
 --

 Key: MAPREDUCE-3410
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3410
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.0
Reporter: Anupam Seth

 If we have an existing queue configuration such as 
 root
  |--- A
  |--- B
 and we attempt to reconfigure it so that we now have
 root
  |--- A
|--- A1
|--- A2
  |--- B
 we get an IOException as follows:
 java.io.IOException: Failed to re-init queues
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:197)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.testInitializeQueue(TestLeafQueue.java:206)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
   at 
 org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:45)
   at 
 org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
   at 
 org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
   at 
 org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
   at 
 org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:172)
   at 
 org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:78)
   at 
 org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:70)
 Caused by: java.io.IOException: Trying to reinitialize root.a from root.a
   at

[jira] [Created] (MAPREDUCE-4437) Race in MR ApplicationMaster can cause reducers to never be scheduled

Jason Lowe created MAPREDUCE-4437:
-

 Summary: Race in MR ApplicationMaster can cause reducers to never 
be scheduled
 Key: MAPREDUCE-4437
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4437
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: applicationmaster, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe


If the MR AM is notified of container completion by the RM before the AM 
receives notification of the container cleanup from the NM then it can fail to 
schedule reducers indefinitely.  Logs showing the issue to follow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4437) Race in MR ApplicationMaster can cause reducers to never be scheduled


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413783#comment-13413783
 ] 

Jason Lowe commented on MAPREDUCE-4437:
---

This is an excerpt of the AM log from a sleep job with 20 mappers and 20 
reducers and mapreduce.job.reduce.slowstart.completedmaps=1.0:

{noformat}
...
2012-07-12 15:51:35,721 INFO [RMCommunicator Allocator] 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed 
container container_1342108144362_0002_01_17
2012-07-12 15:51:35,721 INFO [RMCommunicator Allocator] 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed 
container container_1342108144362_0002_01_15
2012-07-12 15:51:35,721 INFO [RMCommunicator Allocator] 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: 
PendingReduces:20 ScheduledMaps:0 ScheduledReduces:0 AssignedMaps:0 
AssignedReduces:0 completedMaps:18 completedReduces:0 containersAllocated:25 
containersReleased:5 hostLocalAssigned:0 rackLocalAssigned:0 
availableResources(headroom):memory: 112640
2012-07-12 15:51:35,721 INFO [RMCommunicator Allocator] 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating 
schedule...
2012-07-12 15:51:35,721 INFO [RMCommunicator Allocator] 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start 
threshold not met. completedMapsForReduceSlowstart 20
2012-07-12 15:51:35,721 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_14_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,721 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_05_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,721 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_09_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,721 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_07_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,722 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_10_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,722 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_03_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,722 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_18_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,722 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_12_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,722 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_08_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,722 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_02_0: 
2012-07-12 15:51:35,722 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_17_0: 
2012-07-12 15:51:35,722 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_11_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,722 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_01_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,722 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_16_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,722 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report 
from attempt_1342108144362_0002_m_15_0: Container killed by the 
ApplicationMaster.

2012-07-12 15:51:35,723 INFO [AsyncDispatcher event handler]

[jira] [Updated] (MAPREDUCE-4437) Race in MR ApplicationMaster can cause reducers to never be scheduled


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4437:
---

Priority: Critical  (was: Major)

 Race in MR ApplicationMaster can cause reducers to never be scheduled
 -

 Key: MAPREDUCE-4437
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4437
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: applicationmaster, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Priority: Critical

 If the MR AM is notified of container completion by the RM before the AM 
 receives notification of the container cleanup from the NM then it can fail 
 to schedule reducers indefinitely.  Logs showing the issue to follow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-1720) 'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker UI


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J resolved MAPREDUCE-1720.


  Resolution: Not A Problem
Target Version/s:   (was: 0.22.0)

Given that the UI does indicate in the reason if its FAILED or KILLED, and 
MR2's UIs (YARN's and JHS' both) does not have this specific issue anymore 
today, am closing this as Not A Problem. Feel free to reopen and rebase the 
patch if you feel such a change is very worthy on the sustaining MR1 side.

  'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker 
 UI
 

 Key: MAPREDUCE-1720
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1720
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.20.1
 Environment: all
Reporter: Subramaniam Krishnan
Assignee: Harsh J
  Labels: critical-0.22.0
 Attachments: mapred.failed.killed.difference.png, 
 mapreduce.unsuccessfuljobs.ui.r1.diff


 The JobTracker UI shows both Failed/Killed Jobs as Failed. The Killed job 
 status has been separated from Failed as part of HADOOP-3924, so the UI needs 
 to be updated to reflect the same.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413802#comment-13413802
 ] 

Robert Joseph Evans commented on MAPREDUCE-4432:


I am +1 on this patch too.  Thanks Gabriel for helping to make Hadoop better. 

 Confusing warning message when GenericOptionsParser is not used
 ---

 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Gabriel Reid
Priority: Trivial
 Attachments: MAPREDUCE-4432.patch, MAPREDUCE-4432.patch


 The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
 parsing the arguments. Applications should implement Tool for the same. -- 
 is confusing and (probably) grammatically incorrect.
 This can be improved by having an updated warning message which gives clearer 
 directions on what can be improved in the application to avoid the warning in 
 the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-4432:
---

  Resolution: Fixed
   Fix Version/s: 2.0.1-alpha
Target Version/s:   (was: 2.0.1-alpha)
  Status: Resolved  (was: Patch Available)

Committed to branch-2 and trunk. Thanks for the contribution Gabriel, and 
thanks for the additional review too Robert!

 Confusing warning message when GenericOptionsParser is not used
 ---

 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Gabriel Reid
Priority: Trivial
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4432.patch, MAPREDUCE-4432.patch


 The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
 parsing the arguments. Applications should implement Tool for the same. -- 
 is confusing and (probably) grammatically incorrect.
 This can be improved by having an updated warning message which gives clearer 
 directions on what can be improved in the application to avoid the warning in 
 the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413821#comment-13413821
 ] 

Hudson commented on MAPREDUCE-4432:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2529 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2529/])
MAPREDUCE-4432. Confusing warning message when GenericOptionsParser is not 
used. Contributed by Gabriel Reid. (harsh) (Revision 1361273)

 Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361273
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java


 Confusing warning message when GenericOptionsParser is not used
 ---

 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Gabriel Reid
Priority: Trivial
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4432.patch, MAPREDUCE-4432.patch


 The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
 parsing the arguments. Applications should implement Tool for the same. -- 
 is confusing and (probably) grammatically incorrect.
 This can be improved by having an updated warning message which gives clearer 
 directions on what can be improved in the application to avoid the warning in 
 the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413823#comment-13413823
 ] 

Hudson commented on MAPREDUCE-4432:
---

Integrated in Hadoop-Common-trunk-Commit #2463 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2463/])
MAPREDUCE-4432. Confusing warning message when GenericOptionsParser is not 
used. Contributed by Gabriel Reid. (harsh) (Revision 1361273)

 Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361273
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java


 Confusing warning message when GenericOptionsParser is not used
 ---

 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Gabriel Reid
Priority: Trivial
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4432.patch, MAPREDUCE-4432.patch


 The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
 parsing the arguments. Applications should implement Tool for the same. -- 
 is confusing and (probably) grammatically incorrect.
 This can be improved by having an updated warning message which gives clearer 
 directions on what can be improved in the application to avoid the warning in 
 the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4428) A failed job is not available under job history if the job is killed right around the time job is notified as failed

[
https://issues.apache.org/jira/browse/MAPREDUCE-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413840#comment-13413840
]

Robert Joseph Evans commented on MAPREDUCE-4428:

You should not need to restart all of yarn to update the counters max. You
should be able to set it on a per application basis assuming that you do not
have it marked as final in mapred-site.xml, although you may get similar errors
in the History Server if you do that.

Could you please file a separate JIRA for the counter's limit issue. We should
have a cleaner way to deal with the counter's limit being exceeded.

I agree with you that this is a fix that needs to happen, Sadly it is just not
a simple fix. I will talk with some co-workers about this to see that we can
come up with.

A failed job is not available under job history if the job is killed right
around the time job is notified as failed
-

Key: MAPREDUCE-4428
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4428
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, jobtracker
Affects Versions: 2.0.0-alpha
Reporter: Rahul Jain
Attachments: am_failed_counter_limits.txt, appMaster_bad.txt,
appMaster_good.txt, resrcmgr_bad.txt

We have observed this issue consistently running hadoop CDH4 version (based
upon 2.0 alpha release):
In case our hadoop client code gets a notification for a completed job (
using RunningJob object job, with (job.isComplete()
job.isSuccessful()==false)
the hadoop client code does an unconditional job.killJob() to terminate the
job.
With earlier hadoop versions (verified on hadoop 0.20.2 version), we still
have full access to job logs afterwards through hadoop console. However, when
using MapReduceV2, the failed hadoop job no longer shows up under jobhistory
server. Also, the tracking URL of the job still points to the non-existent
Application master http port.
Once we removed the call to job.killJob() for failed jobs from our hadoop
client code, we were able to access the job in job history with mapreduce V2
as well. Therefore this appears to be a race condition in the job management
wrt. job history for failed jobs.
We do have the application master and node manager logs collected for this
scenario if that'll help isolate the problem and the fix better.

[jira] [Commented] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413855#comment-13413855
 ] 

Hudson commented on MAPREDUCE-4432:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #2483 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2483/])
MAPREDUCE-4432. Confusing warning message when GenericOptionsParser is not 
used. Contributed by Gabriel Reid. (harsh) (Revision 1361273)

 Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361273
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java


 Confusing warning message when GenericOptionsParser is not used
 ---

 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Gabriel Reid
Priority: Trivial
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4432.patch, MAPREDUCE-4432.patch


 The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
 parsing the arguments. Applications should implement Tool for the same. -- 
 is confusing and (probably) grammatically incorrect.
 This can be improved by having an updated warning message which gives clearer 
 directions on what can be improved in the application to avoid the warning in 
 the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-4428) A failed job is not available under job history if the job is killed right around the time job is notified as failed


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans reassigned MAPREDUCE-4428:
--

Assignee: Robert Joseph Evans

 A failed job is not available under job history if the job is killed right 
 around the time job is notified as failed 
 -

 Key: MAPREDUCE-4428
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4428
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver, jobtracker
Affects Versions: 2.0.0-alpha
Reporter: Rahul Jain
Assignee: Robert Joseph Evans
 Attachments: am_failed_counter_limits.txt, appMaster_bad.txt, 
 appMaster_good.txt, resrcmgr_bad.txt


 We have observed this issue consistently running hadoop CDH4 version (based 
 upon 2.0 alpha release):
 In case our hadoop client code gets a notification for a completed job ( 
 using RunningJob object job, with (job.isComplete()  
 job.isSuccessful()==false)
 the hadoop client code does an unconditional job.killJob() to terminate the 
 job.
 With earlier hadoop versions (verified on hadoop 0.20.2 version), we still  
 have full access to job logs afterwards through hadoop console. However, when 
 using MapReduceV2, the failed hadoop job no longer shows up under jobhistory 
 server. Also, the tracking URL of the job still points to the non-existent 
 Application master http port.
 Once we removed the call to job.killJob() for failed jobs from our hadoop 
 client code, we were able to access the job in job history with mapreduce V2 
 as well. Therefore this appears to be a race condition in the job management 
 wrt. job history for failed jobs.
 We do have the application master and node manager logs collected for this 
 scenario if that'll help isolate the problem and the fix better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4428) A failed job is not available under job history if the job is killed right around the time job is notified as failed

[
https://issues.apache.org/jira/browse/MAPREDUCE-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413864#comment-13413864
]

Robert Joseph Evans commented on MAPREDUCE-4428:

OK I think I have an idea, but Sid I would like your opinion on this. If you
want to pull in Arun on this too I am happy for his opinion too.

What if we augment the ContainerLaunchContext to have something like a cleanup
on kill boolean and a cleanup on bad exit boolean. If cleanup on kill is set
and the container is forcibly killed or if cleanup on bad exit is set and the
container exits with a non-zero status, the NM would try to rerun the
container, but with an environment variable set saying that it is being rerun
for cleanup. The NM would give it a configurable amount of time, say 20
seconds, to do the cleanup, and then if it has not already exited it will shoot
it.

The RM would need a new variable when the AM is submitted to indicate that this
should happen, and then if that is set it would turn on cleanup on kill for the
AM when it is launched, and it would turn on cleanup on bad exit, when it is
launching the AM for the last retry.

The MR AM would have to be modified to look for the environment variable and
only do cleanup if it sees it. The MR client would have to be modified to set
this boolean variable.

A failed job is not available under job history if the job is killed right
around the time job is notified as failed
-

[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's that are not managed by it


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413875#comment-13413875
 ] 

Bikas Saha commented on MAPREDUCE-4427:
---

The findbugs are from FairScheduler and unrelated to this patch.

 Enable the RM to work with AM's that are not managed by it
 --

 Key: MAPREDUCE-4427
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
  Labels: mrv2
 Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch


 Currently, the RM itself manages the AM by allocating a container for it and 
 negotiating the launch on the NodeManager and manages the AM lifecycle. 
 Thereafter, the AM negotiates resources with the RM and launches tasks to do 
 the real work.
 It would be a useful improvement to enhance this model by allowing the AM to 
 be launched independently by the client without requiring the RM. These AM's 
 would be launched on a gateway machine that can talk to the cluster. This 
 would open up new use cases such as the following
 1) Easy debugging of AM, specially during initial development. Having the AM 
 launched on an arbitrary cluster node makes it hard to looks at logs or 
 attach a debugger to the AM. If it can be launched locally then these tasks 
 would be easier.
 2) Running AM's that need special privileges that may not be available on 
 machines managed by the NodeManager

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4438) Add client side for UnmanagedRM

Bikas Saha created MAPREDUCE-4438:
-

 Summary: Add client side for UnmanagedRM
 Key: MAPREDUCE-4438
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4438
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Bikas Saha




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413878#comment-13413878
 ] 

Arun C Murthy commented on MAPREDUCE-3451:
--

Unfortunately we've introduced findbugs warnings, Alejandro you missed them?

I'll file another jira.

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4438) Add client side for UnmanagedRM


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated MAPREDUCE-4438:
--

Description: MAPREDUCE-4427 added server side support for umanaged AM's. 
This tracks creating client side supporting pieces.
   Assignee: Bikas Saha

 Add client side for UnmanagedRM
 ---

 Key: MAPREDUCE-4438
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4438
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha

 MAPREDUCE-4427 added server side support for umanaged AM's. This tracks 
 creating client side supporting pieces.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4439) MAPREDUCE-3451 introduced a bunch of findbugs warnings

Arun C Murthy created MAPREDUCE-4439:


 Summary: MAPREDUCE-3451 introduced a bunch of findbugs warnings
 Key: MAPREDUCE-4439
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4439
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Arun C Murthy
Priority: Blocker




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4440) Change SchedulerApp to be a minimal interface

Arun C Murthy created MAPREDUCE-4440:


 Summary: Change SchedulerApp to be a minimal interface 
 Key: MAPREDUCE-4440
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4440
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.1-alpha
Reporter: Arun C Murthy
 Fix For: 2.0.1-alpha


Schedulers should manage their own implementations of SchedulerApp.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-4440) Change SchedulerApp to be a minimal interface


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun C Murthy reassigned MAPREDUCE-4440:


Assignee: Arun C Murthy

 Change SchedulerApp to be a minimal interface 
 --

 Key: MAPREDUCE-4440
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4440
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.1-alpha
Reporter: Arun C Murthy
Assignee: Arun C Murthy
 Fix For: 2.0.1-alpha


 Schedulers should manage their own implementations of SchedulerApp.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's that are not managed by it

[
https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413890#comment-13413890
]

Bikas Saha commented on MAPREDUCE-4427:
---

[~alfthomas]
Documentation will come with the client side code that I will imminently add.
Just opened MAPREDUCE-4438 for it.
The idea of queue level enable/disable seemed good but has been kept off for
now because there does not seem to be a strong reason to require explicit
approval. Perhaps you are thinking of some scenario I havent considered. Its
not much work to add it if needed though.
There should be no change needed to an appmaster to be used in an unmanaged
mode. It should be the same as running it on the cluster. Thats is key to
enable the debug/dev-iteration scenario. The client side work will use
distributed shell as an example.

Enable the RM to work with AM's that are not managed by it
--

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413899#comment-13413899
 ] 

Arun C Murthy commented on MAPREDUCE-4049:
--

Also, we should add at least one unit test with an alternate 
ShuffleProvider/ShuffleConsumer to ensure that we fail fast if we break them.

 plugin for generic shuffle service
 --

 Key: MAPREDUCE-4049
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
  Labels: merge, plugin, rdma, shuffle
 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, 
 HADOOP-1.1.patch, HADOOP-1.x.y-review-oriented.patch, Hadoop Shuffle Consumer 
 Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml


 Support generic shuffle service as set of two plugins: ShuffleProvider  
 ShuffleConsumer.
 This will satisfy the following needs:
 # Better shuffle and merge performance. For example: we are working on 
 shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, 
 or Infiniband) instead of using the current HTTP shuffle. Based on the fast 
 RDMA shuffle, the plugin can also utilize a suitable merge approach during 
 the intermediate merges. Hence, getting much better performance.
 # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden 
 dependency of NodeManager with a specific version of mapreduce shuffle 
 (currently targeted to 0.24.0).
 References:
 # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu 
 from Auburn University with others, 
 [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
 # I am attaching 2 documents with suggested Top Level Design for both plugins 
 (currently, based on 1.0 branch)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413898#comment-13413898
]

Arun C Murthy commented on MAPREDUCE-4049:
--

bq. I'll be happy to submit a corresponding patch for hadoop-2. Is it possible
to start with hadoop-1 patch in order to have it on time for 1.1.0? - I'll be
eager to do the hadoop-2 patch immediately after it.

Normally we do both simultaneously. However, given our lack of attention on
this patch I'd be more than willing to ignore that policy, assuming there are
no other objections. OTOH, I'd really appreciate some due-diligence on trunk
too; mainly to ensure we don't break compatibility across the versions. I hope
that is reasonable?

bq. We use taskDone/jobDone as optional messages to the plugin. We have
experimental variant of the plugin that uses these messages for optimizing
cache handling.

Can you share more details? I'm just worried about our ability to continue to
support these while we have no idea what they are used for. Thanks.

plugin for generic shuffle service
--

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413901#comment-13413901
 ] 

Arun C Murthy commented on MAPREDUCE-3451:
--

I'm seeing another problem after I did a 'git pull' today after this commit.

My IDE complains bitterly that TestFSSchdulerApp has the wrong package 
declaration.

[~tucu00]: Should we revert this and fix the package problem and the findbugs 
warnings?

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413903#comment-13413903
 ] 

Arun C Murthy commented on MAPREDUCE-3451:
--

bq. My IDE complains bitterly that TestFSSchdulerApp has the wrong package 
declaration.

Also, my IDE complains that Queue needs an import.

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413905#comment-13413905
 ] 

Arun C Murthy commented on MAPREDUCE-3451:
--

Digging more - looks like I don't see the fair-scheduler tests show up in 
hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar.

Is that the reason why we don't see a compilation failure even though my IDE 
complains?

Some maven work remaining, perhaps?

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413906#comment-13413906
 ] 

Arun C Murthy commented on MAPREDUCE-3451:
--

bq. Digging more - looks like I don't see the fair-scheduler tests show up in 
hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar.

Spoke too soon, I see the following:

{noformat}
$ jar -tvf 
./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar
 | grep -i fair
 0 Fri Jul 13 10:37:20 PDT 2012 
org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/
   324 Fri Jul 13 10:37:20 PDT 2012 
org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler$1.class
  1463 Fri Jul 13 10:37:20 PDT 2012 
org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler$MockClock.class
 25975 Fri Jul 13 10:37:20 PDT 2012 
org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.class
{noformat}

However, I don't see TestFSSchedulerApp, not sure why I don't see a compilation 
failure given the wrong package declaration.

Patrick - any idea?

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4403) Adding test case for resubmission of jobs in TestRecoveryManager


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Bansal updated MAPREDUCE-4403:
-

Issue Type: Improvement  (was: Bug)

 Adding test case for resubmission of jobs in TestRecoveryManager
 

 Key: MAPREDUCE-4403
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4403
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobtracker
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Fix For: 0.22.1

 Attachments: MAPREDUCE-4403-22-1.patch, MAPREDUCE-4403-22.patch


 In Hadoop 22 Test recovery Manager does not have resubmission test case which 
 checks after the resubmission jobs get succeeded.
 There is some refactoring is also needed. 
 Thanks,
 Mayank

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4393) PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS

2012-07-13 Thread Jaigak Song (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413910#comment-13413910
 ] 

Jaigak Song commented on MAPREDUCE-4393:


I just published a document about the prototype and findings which you might 
already know. If you are interested, here is the url: 
http://jaigak.blogspot.com/2012/07/paas-on-hadoop-yarn-idea-and-prototype.html

 PaaS on YARN: an YARN application to demonstrate that YARN can be used as a 
 PaaS
 

 Key: MAPREDUCE-4393
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4393
 Project: Hadoop Map/Reduce
  Issue Type: Task
  Components: examples
Affects Versions: 0.23.1
Reporter: Jaigak Song
Assignee: Jaigak Song
 Fix For: 3.0.0

 Attachments: HADOOPasPAAS_Architecture.pdf, MAPREDUCE-4393.patch, 
 MAPREDUCE-4393.patch, MAPREDUCE-4393.patch, MAPREDUCE4393.patch, 
 MAPREDUCE4393.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 This application is to demonstrate that YARN can be used for non-mapreduce 
 applications. As Hadoop has already been adopted and deployed widely and its 
 deployment in future will be highly increased, we thought that it's a good 
 potential to be used as PaaS.  
 I have implemented a proof of concept to demonstrate that YARN can be used as 
 a PaaS (Platform as a Service). I have done a gap analysis against VMware's 
 Cloud Foundry and tried to achieve as many PaaS functionalities as possible 
 on YARN.
 I'd like to check in this POC as a YARN example application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4403) Adding test case for resubmission of jobs in TestRecoveryManager


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413912#comment-13413912
 ] 

Mayank Bansal commented on MAPREDUCE-4403:
--

This is not applicable for trunk

Thanks,
Mayank

 Adding test case for resubmission of jobs in TestRecoveryManager
 

 Key: MAPREDUCE-4403
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4403
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobtracker
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Fix For: 0.22.1

 Attachments: MAPREDUCE-4403-22-1.patch, MAPREDUCE-4403-22.patch


 In Hadoop 22 Test recovery Manager does not have resubmission test case which 
 checks after the resubmission jobs get succeeded.
 There is some refactoring is also needed. 
 Thanks,
 Mayank

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-4403) Adding test case for resubmission of jobs in TestRecoveryManager


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Bansal resolved MAPREDUCE-4403.
--

Resolution: Fixed

 Adding test case for resubmission of jobs in TestRecoveryManager
 

 Key: MAPREDUCE-4403
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4403
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobtracker
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Fix For: 0.22.1

 Attachments: MAPREDUCE-4403-22-1.patch, MAPREDUCE-4403-22.patch


 In Hadoop 22 Test recovery Manager does not have resubmission test case which 
 checks after the resubmission jobs get succeeded.
 There is some refactoring is also needed. 
 Thanks,
 Mayank

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers

2012-07-13 Thread Andrew Ferguson (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413913#comment-13413913
]

Andrew Ferguson commented on MAPREDUCE-4334:

hi all, I think there are pros and cons to both approaches, which I will try to
outline below.

Cgroups:
- they provide a coherent path for future resource management: network
bandwidth, CPU upper- and lower-bounds, block I/O priorities and limits, etc.
[1]
- can be integrated with resource management for other applications, drawing
upon a single resource budget for a group of users
- cgroup's hierarchies are key to this. in a taskset-only world, the NM would
need to be given a fixed allocation of the node's CPUs to manage
- cgroups are not persistent across reboots. this is unfortunate. however, 1)
anyone using them needs to mount them on startup, so they will need to make a
change to their startup process already, and 2) there are extensive,
cross-distro tools to create and manage cgroups automatically on reboot (RHEL 6
has great docs on them [2])
- some clusters are already using Cgroups, without any support from
Hadoop/YARN. for example, StumbleUpon [3]

Taskset:
- compatible with RHEL 5
- does not require changes to node startup
- can be implemented with a SUID root binary, as LCE is today

My first design for this JIRA had the LCE create the cgroups. This turned out
to be the wrong approach for several reasons:
- What if I wanted to use the regular container executor with cgroups? An admin
may not allow me to have a setuid root binary, but may be willing to create a
cgroup hierarchy for me (after all, this is one advantage of the hierarchy:
delegation)
- Conversely, what if I wanted to use the LCE without cgroups?
- There needs to be a part of the NM responsible for deleting unused cgroups,
and the other tasks of a ContainersManager I described in MAPREDUCE-4351. Some
of those are specific to how resource enforcement is being done; it seemed best
to keep that code together in the ContainersManager, then spread across a
ContainersManager and the LCE.
- Putting the resource enforcement smarts in the ContainersMonitor (which is
already receiving events from the RM), allows it to dynamically adjust the
resource enforcement
- On startup, the JVM can appear to be using twice as much memory as it
actually is (see comment in ContainersMonitorImpl.java). By starting the JVM
within the cgroup, rather than allowing it to start outside the cgroup and
moving it into the cgroup with a ContainersMonitor as my patch above does, the
kernel may kill the JVM inadvertently.

I really like the flexibility of keeping the LCE and resource enforcement
separate. In my mind, the LCE is for starting processes, and should stick to
doing that. Resource enforcement is a separate job.

My recommendation is the following:
1) Keep the LCE as it is.
2) Support pluggable ContainersMonitors (MAPREDUCE-4351)
3) For cgroups, we can start with the patch above. It is best for admins who
already use cgroups on their nodes and want to have YARN take advantage of
them. (This is the point of the yarn.nodemanger.cgroups.path config option I
added)
4) For cgroups, we could have a second ContainersMonitor plugin which uses a
setuid root binary to also mount create cgroups, freeing the admin from
managing them at all.
5) For taskset, we can implement a ContainersMonitor which uses a setuid root
binary (potentially the LCE, but perhaps better if it's something else, just to
keep the security footprint down) to pin processes to CPUs. This
ContainersMonitor will also need the memory enforcement code from the current
ContainersMonitorImpl

I've done 1-3 (well, #1 is a freebie :-) ... and I can definitely do #5 as well.

Arun, does this design appeal to you?

[1]
http://www.linux-kongress.org/2010/slides/seyfried-cgroups-linux-kongress-2010-presentation.pdf
[2]
https://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/index.html
[3] http://inchoate-clatter.blogspot.com/2012/03/hbase-ops-automation.html

Add support for CPU isolation/monitoring of containers
--

Key: MAPREDUCE-4334
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Andrew Ferguson
Attachments: MAPREDUCE-4334-pre1.patch,
MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch,
MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch

Once we get in MAPREDUCE-4327, it will be important to actually enforce
limits on CPU consumption of containers.
Several options spring to mind:
# taskset (RHEL5+)
# cgroups (RHEL6+)

--
This message is automatically generated by JIRA.
If you

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413914#comment-13413914
 ] 

Karthik Kambatla commented on MAPREDUCE-3451:
-

Arun, in the last version of the patch, Patrick moved TestFSSchdulerApp from 
o.a.h.y.s.rm.scheduler to o.a.h.y.s.rm.scheduler.fair. He might have forgotten 
to update the package name accordingly. From what I understand, because of the 
wrong package name, the class file is still created one level above.

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413915#comment-13413915
 ] 

Karthik Kambatla commented on MAPREDUCE-3451:
-

Do you suggest creating a clean-up JIRA and addressing these issues?

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's that are not managed by it

[
https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413916#comment-13413916
]

Arun C Murthy commented on MAPREDUCE-4427:
--

bq. Is there a way to make sure folks can't use this if admins chose?

Interesting point. I'd like to think through that requirement before adding it
in - what were you thinking Thomas?

Maybe we can add that as an enhancement via a separate jira?

Enable the RM to work with AM's that are not managed by it
--

[jira] [Updated] (MAPREDUCE-4405) Adding test case for HierarchicalQueue in TestJobQueueClient


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Bansal updated MAPREDUCE-4405:
-

Attachment: MAPREDUCE-4405-22-v2.patch

Incorporating Konstantin's comment

Thanks,
Mayank

 Adding test case for HierarchicalQueue in TestJobQueueClient
 

 Key: MAPREDUCE-4405
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4405
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Attachments: MAPREDUCE-4405-22-v2.patch, MAPREDUCE-4405-22.patch


 Adding test case for HierarchicalQueue in TestJobQueueClient

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4405) Adding test case for HierarchicalQueue in TestJobQueueClient


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413948#comment-13413948
 ] 

Konstantin Shvachko commented on MAPREDUCE-4405:


+1 looks good.

 Adding test case for HierarchicalQueue in TestJobQueueClient
 

 Key: MAPREDUCE-4405
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4405
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: client
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Attachments: MAPREDUCE-4405-22-v2.patch, MAPREDUCE-4405-22.patch


 Adding test case for HierarchicalQueue in TestJobQueueClient

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4405) Adding test case for HierarchicalQueue in TestJobQueueClient


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko updated MAPREDUCE-4405:
---

Fix Version/s: 0.22.1
 Hadoop Flags: Reviewed

I just committed this to branch 0.22.1. Thank you Mayank.

 Adding test case for HierarchicalQueue in TestJobQueueClient
 

 Key: MAPREDUCE-4405
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4405
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: client
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Fix For: 0.22.1

 Attachments: MAPREDUCE-4405-22-v2.patch, MAPREDUCE-4405-22.patch


 Adding test case for HierarchicalQueue in TestJobQueueClient

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4415) Backport the Job.getInstance methods from MAPREDUCE-1505 to branch-1


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413962#comment-13413962
 ] 

Harsh J commented on MAPREDUCE-4415:


Arun - Ping?

 Backport the Job.getInstance methods from MAPREDUCE-1505 to branch-1
 

 Key: MAPREDUCE-4415
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4415
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 1.0.0
Reporter: Harsh J
Assignee: Harsh J
 Attachments: MAPREDUCE-4415.patch


 In 2.x MR, the Job constructors have all been deprecated in favor of 
 Job.getInstance() calls to get a Job object.
 However, these getInstance methods do not appear to be present in the 1.x MR 
 API, and thereby may cause additional pain to users moving from 1.x to 2.x 
 going forward.
 This patch proposes to add in the getInstance style of methods with suitable 
 test coverage for both style of constructors, while not pulling in anything 
 else from MAPREDUCE-1505 (as we lack 'Cluster' in 1.x). As we're not going to 
 be deprecating the regular ctors in a 1.x release, this is not an 
 incompatible change in any way.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration

2012-07-13 Thread Ahmed Radwan (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413970#comment-13413970
 ] 

Ahmed Radwan commented on MAPREDUCE-4422:
-

Thanks Arun for the clarifications! I have already submitted a new patch 
yesterday incorporating your comments. Please let me know if you have any other 
comments. 

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch, 
 MAPREDUCE-4422_rev3.patch, MAPREDUCE-4422_rev3.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4395) Possible NPE at ClientDistributedCacheManager#determineTimestamps


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413979#comment-13413979
 ] 

Robert Joseph Evans commented on MAPREDUCE-4395:


Looking at the patch it looks OK to me.  I looked and the only other thing that 
uses this API is streaming, when it is setting up a JobConf, so I am OK with it 
blowing up when the URI is not valid.  My only comment is that I would like the 
Javadocs for this method updated to explain what it does and also indicate what 
happens in the failure case.  because well

{code}
   /**
*
* @param str
*/
{code}

is completely useless.  

 Possible NPE at ClientDistributedCacheManager#determineTimestamps
 -

 Key: MAPREDUCE-4395
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4395
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: distributed-cache, job submission, mrv2
Affects Versions: 2.0.0-alpha, trunk
Reporter: Bhallamudi Venkata Siva Kamesh
Assignee: Bhallamudi Venkata Siva Kamesh
Priority: Critical
 Attachments: MAPREDUCE-4395.patch


 {code:title=ClientDistributedCacheManager#determineTimestamps|borderStyle=solid}
 URI[] tfiles = DistributedCache.getCacheFiles(job);
 {code}
 It may be possible that tfiles array contains *null* as it's entry, and 
 subsequently leads to NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4157) ResourceManager should not kill apps that are well behaved


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413994#comment-13413994
 ] 

Robert Joseph Evans commented on MAPREDUCE-4157:


I looked through the new patch and I am still a +1 on this change.

 ResourceManager should not kill apps that are well behaved
 --

 Key: MAPREDUCE-4157
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4157
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 2.0.0-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
 Attachments: MAPREDUCE-4157.patch, MAPREDUCE-4157.patch


 Currently when the ApplicationMaster unregisters with the ResourceManager, 
 the RM kills (via the NMs) all the active containers for an application.  
 This introduces a race where the AM may be trying to clean up and may not 
 finish before it is killed.  The RM should give the AM a chance to exit 
 cleanly on its own rather than always race with a pending kill on shutdown.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2

2012-07-13 Thread Patrick Wendell (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413997#comment-13413997
 ] 

Patrick Wendell commented on MAPREDUCE-3451:


Yes the error is related to the last minute movement of that test into the Fair 
package.

Karthik - could you quickly patch this so we can get the build stabilized? If 
you can fix the findbugs quickly that would be great too - as i said earlier 
these are false positives but we might be able to coerce findbugs into not 
spouting warnings.

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2

2012-07-13 Thread Patrick Wendell (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413998#comment-13413998
 ] 

Patrick Wendell commented on MAPREDUCE-3451:


It's a one line change to the package header. The findbugs issues are discussed 
further up in this Jira and were also discussed during the last round of 
reviews.

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4441) Fix build issue caused by MR-3451

Karthik Kambatla created MAPREDUCE-4441:
---

 Summary: Fix build issue caused by MR-3451
 Key: MAPREDUCE-4441
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4441
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Karthik Kambatla
 Attachments: MR-3451-build-fix.patch

TestFSSchedulerApp is in the wrong package and missing some imports.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-4441) Fix build issue caused by MR-3451


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla reassigned MAPREDUCE-4441:
---

Assignee: Karthik Kambatla

 Fix build issue caused by MR-3451
 -

 Key: MAPREDUCE-4441
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4441
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: MR-3451-build-fix.patch


 TestFSSchedulerApp is in the wrong package and missing some imports.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4441) Fix build issue caused by MR-3451


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4441:


Attachment: MR-3451-build-fix.patch

Uploading a patch to fix FS build issue introduced by MR-3451.

 Fix build issue caused by MR-3451
 -

 Key: MAPREDUCE-4441
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4441
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Karthik Kambatla
 Attachments: MR-3451-build-fix.patch


 TestFSSchedulerApp is in the wrong package and missing some imports.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4441) Fix build issue caused by MR-3451


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4441:


Fix Version/s: 2.0.1-alpha
Affects Version/s: 2.0.0-alpha
   Status: Patch Available  (was: Open)

 Fix build issue caused by MR-3451
 -

 Key: MAPREDUCE-4441
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4441
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 2.0.1-alpha

 Attachments: MR-3451-build-fix.patch


 TestFSSchedulerApp is in the wrong package and missing some imports.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4283) Display tail of aggregated logs by default


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13414011#comment-13414011
 ] 

Robert Joseph Evans commented on MAPREDUCE-4283:


Jason,

There are some issues with how you seek in the file.

InputStream.available is only supposed to return the value that is left and 
will not block.  It looks like you are using it to try and read to the end of 
the BoundedInputStream, but I am not sure that it is guaranteed to work that 
way.

 Display tail of aggregated logs by default
 --

 Key: MAPREDUCE-4283
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4283
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
 Attachments: MAPREDUCE-4283.patch


 Similar to the manner in which the nodemanager webUI displays container logs, 
 it would be very useful if the historyserver showed the trailing 4K or so of 
 the aggregated logs with a link to see the full log.
 When debugging issues the relevant errors are usually at the end of the log, 
 so showing just the last few K can enable quick diagnosis without waiting for 
 what can be many megabytes of log data to download. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4442) Accessing hadoop counters from a job is unreliable in yarn during in AM process cleanup window

2012-07-13 Thread Rahul Jain (JIRA)

Rahul Jain created MAPREDUCE-4442:
-

 Summary: Accessing hadoop counters from a job is unreliable in 
yarn during in AM process cleanup  window
 Key: MAPREDUCE-4442
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4442
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Rahul Jain


We found this issue during our tests moving from MapReduceV1 to MapReduceV2. A 
few of our applications access job counters multiple times:

a) After submission of job, while job is execution (works fine)

b) Right after job complete notification is received (works fine)

c) Few seconds after job complete notification (fails most of the times).

The error snippet is as follows:

{code}
2012-07-12 19:12:29,039 WARN  [Client] Unexpected error reading responses on 
connection Thread[IPC Client (1252749669) connection to 
sjc1-ciq-ibm-grid07.carrieriq.com/10.202.50.187:47944 from hadoop,5,main]
java.lang.NullPointerException
at 
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781)
2012-07-12 19:12:29,044 INFO  [ClientServiceDelegate] Application state is 
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-07-12 19:12:29,132 INFO  [ClientServiceDelegate] Application state is 
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2012-07-12 19:12:29,216 ERROR [UserGroupInformation] PriviledgedActionException 
as:hadoop (auth:SIMPLE) cause:java.io.IOException
2012-07-12 19:12:29,216 WARN  [BaseOutputStageJob] getJobCounters: Unable to 
retrieve counters. null
java.io.IOException
at 
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:315)
at 
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:335)
at 
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:470)
at org.apache.hadoop.mapreduce.Job$8.run(Job.java:719)
at org.apache.hadoop.mapreduce.Job$8.run(Job.java:716)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:716)
at 
org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:396)
{code}

 
I'll attach the logs for AM and resource mgr separately, however no unusual 
activity is seen in those.

This makes me suspect that we have a race condition in the code trying to 
access job counters when AM is finishing up and the job hasn't moved to history 
server yet.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's that are not managed by it

2012-07-13 Thread Thomas Graves (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13414014#comment-13414014
]

Thomas Graves commented on MAPREDUCE-4427:
--

note - I agree this is good stuff!

Sorry I meant add it to the client side of the app - which you've answered,
thanks.

I was thinking of any multi-tenant cluster, where people could start abusing
the option and running things on gateway machines and overloading those
gateways or perhaps just causing traffic between outside machines that SEs
don't want or expect. But thinking about this more, there are plenty of other
ways to cause issues like that so I'm good with leaving this off. If a use case
ever comes up we can revisit.

Another question or atleast something to think about - it appears it still goes
through all the queue checks when submitting the application. I'm wondering if
some of those checks might not apply in this case - for instance max am
resources (maxActiveApplications) doesn't really apply because external AM's
aren't using queue capacity for the AM itself. That might not be a big issue
right now if people use this just for debug, but if this is used say by AM's to
launch other AM's in arbitrary containers it might be more of an issue.

What happens when you kill one of these applications? The RM can't really
force kill it - so does it just kill all containers its requested and block
the AM from communicating.

minor nitpicky comments to consider:
Can we just remove the commented out code in the container executor: +
//lfs.mkdir(containerDir, null, false);
Could you also clarify what the comment there means + // Without this app
with no cache files cannot launch tasks? Is it supposed to be without this,
app with And then is this passing true into lfs.mkdir as last parameter?

Enable the RM to work with AM's that are not managed by it
--

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13414015#comment-13414015
 ] 

Arun C Murthy commented on MAPREDUCE-3451:
--

Patrick - the last comment/advice was to supress them (from Harsh). Why weren't 
they?

Now, all patch builds are failing complaining about the findbugs warnings... 

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4428) A failed job is not available under job history if the job is killed right around the time job is notified as failed

2012-07-13 Thread Rahul Jain (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13414017#comment-13414017
 ] 

Rahul Jain commented on MAPREDUCE-4428:
---

OK, will create a separate one for the counter limit exceeded issue.

BTW, I did open MAPREDUCE-4442 for a related issue: we are unable to access job 
counters for the period AM is possibly shutting down as well, it may be a good 
idea to consider that issue in the final fix. 

 A failed job is not available under job history if the job is killed right 
 around the time job is notified as failed 
 -

 Key: MAPREDUCE-4428
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4428
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver, jobtracker
Affects Versions: 2.0.0-alpha
Reporter: Rahul Jain
Assignee: Robert Joseph Evans
 Attachments: am_failed_counter_limits.txt, appMaster_bad.txt, 
 appMaster_good.txt, resrcmgr_bad.txt


 We have observed this issue consistently running hadoop CDH4 version (based 
 upon 2.0 alpha release):
 In case our hadoop client code gets a notification for a completed job ( 
 using RunningJob object job, with (job.isComplete()  
 job.isSuccessful()==false)
 the hadoop client code does an unconditional job.killJob() to terminate the 
 job.
 With earlier hadoop versions (verified on hadoop 0.20.2 version), we still  
 have full access to job logs afterwards through hadoop console. However, when 
 using MapReduceV2, the failed hadoop job no longer shows up under jobhistory 
 server. Also, the tracking URL of the job still points to the non-existent 
 Application master http port.
 Once we removed the call to job.killJob() for failed jobs from our hadoop 
 client code, we were able to access the job in job history with mapreduce V2 
 as well. Therefore this appears to be a race condition in the job management 
 wrt. job history for failed jobs.
 We do have the application master and node manager logs collected for this 
 scenario if that'll help isolate the problem and the fix better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4441) Fix build issue caused by MR-3451

2012-07-13 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13414016#comment-13414016
]

Hadoop QA commented on MAPREDUCE-4441:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12536446/MR-3451-build-fix.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

-1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2587//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2587//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2587//console

This message is automatically generated.

Fix build issue caused by MR-3451
-

Key: MAPREDUCE-4441
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4441
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Fix For: 2.0.1-alpha

Attachments: MR-3451-build-fix.patch

TestFSSchedulerApp is in the wrong package and missing some imports.

[jira] [Updated] (MAPREDUCE-4441) Fix build issue caused by MR-3451