date:20120409


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4019:
-

Priority: Blocker  (was: Minor)

 -list-attempt-ids  is not working
 -

 Key: MAPREDUCE-4019
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.23.1, 0.23.2
Reporter: B Anil Kumar
Priority: Blocker
 Attachments: MAPREDUCE-4019.patch


 while executing {noformat}bin/mapred  job -list-attempt-ids job_id map 
 running{noformat}, we are getting IllegalArgumentexception.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4019) -list-attempt-ids is not working

2012-04-09 Thread Devaraj K (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249777#comment-13249777
 ] 

Devaraj K commented on MAPREDUCE-4019:
--

Thanks Anil for the patch.

Patch mostly looks good to me except few minor nits/doubts on the patch.

- Can we show the states as MAP, REDUCE in the usage info instead of having 
taskTypes map(likewise before mrv2 changes)?
- As of now there is no support for JOB_SETUP, JOB_CLEANUP. Do we need to 
display these also?

 -list-attempt-ids  is not working
 -

 Key: MAPREDUCE-4019
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.23.1, 0.23.2
Reporter: B Anil Kumar
Priority: Blocker
 Attachments: MAPREDUCE-4019.patch


 while executing {noformat}bin/mapred  job -list-attempt-ids job_id map 
 running{noformat}, we are getting IllegalArgumentexception.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4124) Job Priority is not changing

2012-04-09 Thread Nishan Shetty (Created) (JIRA)

Job Priority is not changing 
-

 Key: MAPREDUCE-4124
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4124
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 3.0.0
Reporter: Nishan Shetty
Priority: Critical


1.Submit job
2.Change the job priority using setPriority() or CLI command ./mapred 
job-set-priority job-id priority

Observe that Job priority is not changed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4125) If we are navigating to Nodemanager UI from Resourcemanager,then there is not link to navigate back to Resource manager

2012-04-09 Thread Ramgopal N (Created) (JIRA)

If we are navigating to Nodemanager UI from Resourcemanager,then there is not 
link to navigate back to Resource manager
---

 Key: MAPREDUCE-4125
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4125
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Reporter: Ramgopal N
Priority: Minor


If we are navigating to Nodemanager by clicking on the node link in RM,there is 
no link provided on the NM to navigate back to RM.
 If there is a link to navigate back to RM it would be good

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4126) This is a fix both for oracle support and managing DB splits in general.

2012-04-09 Thread Joseph Doss (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Doss updated MAPREDUCE-4126:
---

Attachment: hadoop.dbinputformat.query.split.patch.patch

 This is a fix both for oracle support and managing DB splits in general.
 

 Key: MAPREDUCE-4126
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4126
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: performance
Affects Versions: 1.0.0
 Environment: Redhat 6.1 x86_64 
Reporter: Joseph Doss
  Labels: hadoop, newbie, patch
 Fix For: 1.0.0

 Attachments: hadoop.dbinputformat.query.split.patch.patch


 This is and addition to an older patch that addressed oracle support. We 
 found that in large result sets, the map reduce was both splitting the 
 results ( as expected ) and running a duplicate ( full result set ) on one 
 node, resulting in a long runtime that should have been reduced by splitting. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4126) This is a fix both for oracle support and managing DB splits in general.

2012-04-09 Thread Joseph Doss (Created) (JIRA)

This is a fix both for oracle support and managing DB splits in general.


 Key: MAPREDUCE-4126
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4126
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: performance
Affects Versions: 1.0.0
 Environment: Redhat 6.1 x86_64 
Reporter: Joseph Doss
 Fix For: 1.0.0
 Attachments: hadoop.dbinputformat.query.split.patch.patch

This is and addition to an older patch that addressed oracle support. We found 
that in large result sets, the map reduce was both splitting the results ( as 
expected ) and running a duplicate ( full result set ) on one node, resulting 
in a long runtime that should have been reduced by splitting. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy

2012-04-09 Thread Robert Joseph Evans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-3921:
---

Status: Patch Available  (was: Open)

Kicking Jenkins.

 MR AM should act on the nodes liveliness information when nodes go 
 up/down/unhealthy
 

 Key: MAPREDUCE-3921
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am, mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Bikas Saha
 Fix For: 0.23.2

 Attachments: MAPREDUCE-3921-1.patch, 
 MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, 
 MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4099) ApplicationMaster may fail to remove staging directory


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-4099:
--

Status: Open  (was: Patch Available)

 ApplicationMaster may fail to remove staging directory
 --

 Key: MAPREDUCE-4099
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4099
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
 Attachments: MAPREDUCE-4099.patch


 When the ApplicationMaster shuts down it's supposed to remove the staging 
 directory, assuming properties weren't set to override this behavior. During 
 shutdown the AM tells the ResourceManager that it has finished before it 
 cleans up the staging directory.  However upon hearing the AM has finished, 
 the RM turns right around and kills the AM container.  If the AM is too slow, 
 the AM will be killed before the staging directory is removed.
 We're seeing the AM lose this race fairly consistently on our clusters, and 
 the lack of staging directory cleanup quickly leads to filesystem quota 
 issues for some users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4099) ApplicationMaster may fail to remove staging directory

2012-04-09 Thread Robert Joseph Evans (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Lowe updated MAPREDUCE-4099:
--

Status: Patch Available (was: Open)

Didn't see any actual failures in the Jenkins build, so maybe it was a timeout?
Tried running the jobclient tests on trunk with and without this patch, and I
didn't see any noticeable time difference. Kicking Jenkins again.

ApplicationMaster may fail to remove staging directory
--

Key: MAPREDUCE-4099
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4099
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Attachments: MAPREDUCE-4099.patch

When the ApplicationMaster shuts down it's supposed to remove the staging
directory, assuming properties weren't set to override this behavior. During
shutdown the AM tells the ResourceManager that it has finished before it
cleans up the staging directory. However upon hearing the AM has finished,
the RM turns right around and kills the AM container. If the AM is too slow,
the AM will be killed before the staging directory is removed.
We're seeing the AM lose this race fairly consistently on our clusters, and
the lack of staging directory cleanup quickly leads to filesystem quota
issues for some users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249844#comment-13249844
 ] 

Robert Joseph Evans commented on MAPREDUCE-3921:


I did a quick look at the code and it looks good to me.  As for the recovery 
error you discovered could you please file a follow up JIRA for it, as it is a 
preexisting issue that can be caused by AM recovery with speculative execution. 
 This patch may expose the issue more frequently, but not enough to really 
worry me that much.  You need two nodes going down very close to one another 
which is possible, but not that often.

 MR AM should act on the nodes liveliness information when nodes go 
 up/down/unhealthy
 

 Key: MAPREDUCE-3921
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am, mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Bikas Saha
 Fix For: 0.23.2

 Attachments: MAPREDUCE-3921-1.patch, 
 MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, 
 MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4117) mapred job -status throws NullPointerException


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4117:
-

Attachment: MAPREDUCE-4117.patch

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4117) mapred job -status throws NullPointerException

2012-04-09 Thread Robert Joseph Evans (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4117:
-

Status: Patch Available  (was: Open)

I have attached patch to fix this issue.

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.

2012-04-09 Thread Robert Joseph Evans (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Joseph Evans updated MAPREDUCE-4079:
---

Attachment: MR-4079-trunk.txt
MR-4079-full-branch-0.23.txt

Up-merged and addressed some issues pointed out in HADOOP-8227.

Allow MR AppMaster to limit ephemeral port range.
-

Key: MAPREDUCE-4079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mr-am, mrv2
Affects Versions: 0.23.2, 2.0.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Blocker
Attachments: MR-4079-full-branch-0.23.txt,
MR-4079-full-branch-0.23.txt, MR-4079-trunk.txt, MR-4079-trunk.txt

Having the MapReduce Application Masters bind to any ephemeral port makes it
very difficult to setup ACLs. mapreduce.job.am-access-disabled from
MAPREDUCE-3251 is not a practical permanent solution for all jobs.
Especially for tools like pig where they are not aware of
mapreduce.job.am-access-disabled and may deal with it properly.
We should add in a config option that would allow someone to restrict the
range of ports that the MR-AM can bind to. It will slow down startup in some
cases because we will have to probe for open ports instead of just asking the
OS to find one for us. But we can make that conditional on this config so
users who do not set this config do not see any performance degradation.

[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249867#comment-13249867
 ] 

Robert Joseph Evans commented on MAPREDUCE-3921:


Someone pointed out to me that my comment is a bit confusing.  When I said two 
nodes going down very close to one another I meant that for this to happen we 
would need one node to go down in succession that had the correct processes 
running on them.   But now that I think about it more, I am not even sure if it 
will expose the issue.

 MR AM should act on the nodes liveliness information when nodes go 
 up/down/unhealthy
 

 Key: MAPREDUCE-3921
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am, mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Bikas Saha
 Fix For: 0.23.2

 Attachments: MAPREDUCE-3921-1.patch, 
 MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, 
 MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4117) mapred job -status throws NullPointerException

2012-04-09 Thread Jonathan Eagles (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249868#comment-13249868
 ] 

Jonathan Eagles commented on MAPREDUCE-4117:


Thanks for the patch Devaraj. One case I'm wondering about is when 
getTaskFailureEventString is called on a Job with only 
non-TaskCompletionEvent.Status.FAILED events. Seems like lastEvent will still 
be null in that case.

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4117) mapred job -status throws NullPointerException

2012-04-09 Thread Robert Joseph Evans (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249869#comment-13249869
]

Hadoop QA commented on MAPREDUCE-4117:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521957/MAPREDUCE-4117.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService

org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry

org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization

org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2173//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2173//console

This message is automatically generated.

mapred job -status throws NullPointerException
--

Key: MAPREDUCE-4117
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
Attachments: MAPREDUCE-4117.patch

{code:xml}
dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status
job_1333408894669_0001
Exception in thread main java.lang.NullPointerException
at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
at java.lang.String.valueOf(String.java:2838)
at java.io.PrintStream.println(PrintStream.java:788)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
{code}

[jira] [Updated] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface

2012-04-09 Thread Robert Joseph Evans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4059:
---

Status: Open  (was: Patch Available)

Up-merging patch.

 The history server should have a separate pluggable storage/query interface
 ---

 Key: MAPREDUCE-4059
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 0.24.0, 0.23.3
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
 Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, 
 MR-4059.txt, MR-4059.txt


 The history server currently caches all parsed jobs in RAM.  These jobs can 
 be very large because of counters.  It would be nice to have a pluggable 
 interface for the cacheing and querying of the cached data so that we can 
 play around with different implementations.  Also just for cleanness of the 
 code it would be nice to split the very large JobHistoryServer.java into a 
 few smaller ones that are more understandable and readable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface

2012-04-09 Thread Robert Joseph Evans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4059:
---

Status: Patch Available  (was: Open)

 The history server should have a separate pluggable storage/query interface
 ---

 Key: MAPREDUCE-4059
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 0.24.0, 0.23.3
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
 Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, 
 MR-4059.txt, MR-4059.txt


 The history server currently caches all parsed jobs in RAM.  These jobs can 
 be very large because of counters.  It would be nice to have a pluggable 
 interface for the cacheing and querying of the cached data so that we can 
 play around with different implementations.  Also just for cleanness of the 
 code it would be nice to split the very large JobHistoryServer.java into a 
 few smaller ones that are more understandable and readable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface

2012-04-09 Thread Robert Joseph Evans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4059:
---

Attachment: MR-4059.txt

No code changes just upmerged.

 The history server should have a separate pluggable storage/query interface
 ---

 Key: MAPREDUCE-4059
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 0.24.0, 0.23.3
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
 Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, 
 MR-4059.txt, MR-4059.txt


 The history server currently caches all parsed jobs in RAM.  These jobs can 
 be very large because of counters.  It would be nice to have a pluggable 
 interface for the cacheing and querying of the cached data so that we can 
 play around with different implementations.  Also just for cleanness of the 
 code it would be nice to split the very large JobHistoryServer.java into a 
 few smaller ones that are more understandable and readable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4127) Resource manager UI does not show the Job Priority

2012-04-09 Thread Nishan Shetty (Created) (JIRA)

Resource manager UI does not show the Job Priority
--

 Key: MAPREDUCE-4127
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4127
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Nishan Shetty


In RM UI the priority of job is not displayed

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3932) MR tasks failing and crashing the AM when available-resources/headRoom becomes zero


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249877#comment-13249877
 ] 

Robert Joseph Evans commented on MAPREDUCE-3932:


@Vinod I know you are swamped is it OK with you if I take this one?

 MR tasks failing and crashing the AM when available-resources/headRoom 
 becomes zero
 ---

 Key: MAPREDUCE-3932
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3932
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am, mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
Priority: Critical
 Fix For: 0.23.2


 [~karams] reported this offline. One reduce task gets preempted because of 
 zero headRoom and crashes the AM.
 {code}
 2012-02-23 11:30:15,956 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: 
 PendingReduces:377 ScheduledMaps:6 ScheduledReduces:23 AssignedMaps:0 
 AssignedReduces:0 completedMaps:4 completedReduces:0 containersAllocated:4 
 containersReleased:0 hostLocalAssigned:0 rackLocalAssigned:4 
 availableResources(headroom):memory: 44544
 2012-02-23 11:30:16,959 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before 
 Scheduling: PendingReduces:377 ScheduledMaps:6 ScheduledReduces:23 
 AssignedMaps:0 AssignedReduces:0 completedMaps:4 completedReduces:0 
 containersAllocated:4 containersReleased:0 hostLocalAssigned:0 
 rackLocalAssigned:4 availableResources(headroom):memory: 44544
 2012-02-23 11:30:16,965 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: 
 PendingReduces:377 ScheduledMaps:6 ScheduledReduces:23 AssignedMaps:0 
 AssignedReduces:0 completedMaps:4 completedReduces:0 containersAllocated:4 
 containersReleased:0 hostLocalAssigned:0 rackLocalAssigned:4 
 availableResources(headroom):memory: 0
 2012-02-23 11:30:16,965 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Assign: 
 PendingReduces:377 ScheduledMaps:6 ScheduledReduces:23 AssignedMaps:0 
 AssignedReduces:0 completedMaps:4 completedReduces:0 containersAllocated:4 
 containersReleased:0 hostLocalAssigned:0 rackLocalAssigned:4 
 availableResources(headroom):memory: 0
 2012-02-23 11:30:16,965 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated 
 containers 3
 2012-02-23 11:30:16,965 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce
 2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned 
 container container_1329995034628_0983_01_06 to 
 attempt_1329995034628_0983_r_00_0
 2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce
 2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned 
 container container_1329995034628_0983_01_07 to 
 attempt_1329995034628_0983_r_01_0
 2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce
 2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned 
 container container_1329995034628_0983_01_08 to 
 attempt_1329995034628_0983_r_02_0
 2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Assign: 
 PendingReduces:377 ScheduledMaps:6 ScheduledReduces:20 AssignedMaps:0 
 AssignedReduces:3 completedMaps:4 completedReduces:0 containersAllocated:7 
 containersReleased:0 hostLocalAssigned:0 rackLocalAssigned:4 
 availableResources(headroom):memory: 0
 2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all 
 scheduled reduces:20
 2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 2
 2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting 
 attempt_1329995034628_0983_r_02_0
 2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting 
 attempt_1329995034628_0983_r_01_0
 2012-02-23 11:30:16,966 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:

[jira] [Commented] (MAPREDUCE-4117) mapred job -status throws NullPointerException

2012-04-09 Thread Devaraj K (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249888#comment-13249888
 ] 

Devaraj K commented on MAPREDUCE-4117:
--

Thanks Bobby for taking a look into the patch.

When the Job is failed and it doesn't contain any task attempts(i.e Job is 
failed before launching any tasks itself) then the lastEvent will be null and 
it will lead to NPE.

 If the Job fails and job contains some attempts means the job should have 
failed because of task failures. This case lastEvent will not be null.

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4117) mapred job -status throws NullPointerException

2012-04-09 Thread Devaraj K (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249891#comment-13249891
 ] 

Devaraj K commented on MAPREDUCE-4117:
--

{quote}
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
{quote}

These test failures are not related to the patch.

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface

2012-04-09 Thread Robert Joseph Evans (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249895#comment-13249895
]

Hadoop QA commented on MAPREDUCE-4059:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521964/MR-4059.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 7 new or modified test
files.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService

org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry

org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization

org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2174//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2174//console

This message is automatically generated.

The history server should have a separate pluggable storage/query interface
---

Key: MAPREDUCE-4059
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.24.0, 0.23.3
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt,
MR-4059.txt, MR-4059.txt

The history server currently caches all parsed jobs in RAM. These jobs can
be very large because of counters. It would be nice to have a pluggable
interface for the cacheing and querying of the cached data so that we can
play around with different implementations. Also just for cleanness of the
code it would be nice to split the very large JobHistoryServer.java into a
few smaller ones that are more understandable and readable.

[jira] [Commented] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249905#comment-13249905
 ] 

Robert Joseph Evans commented on MAPREDUCE-4059:


{quote}org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs{quote}

These tests have been failing randomly and are not related to the patch.

 The history server should have a separate pluggable storage/query interface
 ---

 Key: MAPREDUCE-4059
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 0.24.0, 0.23.3
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
 Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, 
 MR-4059.txt, MR-4059.txt


 The history server currently caches all parsed jobs in RAM.  These jobs can 
 be very large because of counters.  It would be nice to have a pluggable 
 interface for the cacheing and querying of the cached data so that we can 
 play around with different implementations.  Also just for cleanness of the 
 code it would be nice to split the very large JobHistoryServer.java into a 
 few smaller ones that are more understandable and readable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4105) Yarn RackResolver ignores rack configurations

2012-04-09 Thread Tom White (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249909#comment-13249909
 ] 

Tom White commented on MAPREDUCE-4105:
--

+1 thanks for addressing my comments, Ahmed.

 Yarn RackResolver ignores rack configurations
 -

 Key: MAPREDUCE-4105
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4105
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Fix For: 0.23.2

 Attachments: MAPREDUCE-4105.patch, MAPREDUCE-4105_rev2.patch, 
 MAPREDUCE-4105_rev3.patch


 Incorrect mappings because the Yarn RackResolver ignores rack configurations. 
 This can be verified by inspecting the resource manager web ui that lists all 
 the nodes, all of them show up with /default-rack regardless of the output 
 from the script specified using net.topology.script.file.name configuration 
 property.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4117) mapred job -status throws NullPointerException

2012-04-09 Thread Jonathan Eagles (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249989#comment-13249989
 ] 

Jonathan Eagles commented on MAPREDUCE-4117:


My initial thought was the same as yours. However, in TypeConverter, you will 
see that both JobState.FAILED and JobState.ERROR are translated to 
JobStatus.State.FAILED. I believe it to be to much to assume that if 
JobStatus.State is FAILED, then you will be guaranteed to have either 1)a 
failed task completion event or 2) no task completion events at all. It is much 
more clear and simple to me to check for null in this case.

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4128) AM Recovery expects all attempts of a completed task to also be completed.

2012-04-09 Thread Bikas Saha (Created) (JIRA)

AM Recovery expects all attempts of a completed task to also be completed.
--

 Key: MAPREDUCE-4128
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4128
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
 Fix For: 3.0.0


The AM seems to assume that all attempts of a completed task (from a previous 
AM incarnation) would also be completed. There is at least one case in which 
this does not hold. Case being cancellation of a completed task resulting in a 
new running attempt.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4117) mapred job -status throws NullPointerException


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4117:
-

Attachment: MAPREDUCE-4117-1.patch

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117-1.patch, MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4117) mapred job -status throws NullPointerException


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4117:
-

Status: Patch Available  (was: Open)

I am sorry Jon. It's my bad, I have mentioned the name wrongly in the above 
comment.

Thanks for looking and identifying all the cases. I have attached the patch as 
per your suggestion.

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117-1.patch, MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4117) mapred job -status throws NullPointerException


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4117:
-

Status: Open  (was: Patch Available)

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117-1.patch, MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4128) AM Recovery expects all attempts of a completed task to also be completed.

[
https://issues.apache.org/jira/browse/MAPREDUCE-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250010#comment-13250010
]

Bikas Saha commented on MAPREDUCE-4128:
---

In recovery mode, the recovery service assigns a success status to any task
that has a FINISHED event reported. The only way that status could be changed
is if there is a FAILED event for that task, in which case a failed status is
assigned to that task. So once a task is marked with a success status, it
remains so even when subsequent events kill the successful task attempt and
marks it invalid.
Next the recovery service adds all success status tasks into a completedTasks
collection. Then it proceeds to enumerate the events and process them. When it
hits a TaskEventType.*_KILLED/FAILED/SUCCEEDED then it removes those attempts
from the completedTasks. Recovery does not complete until all attempts of all
completedTasks are removed. Now the following sequence of events can happen for
Tasks A and B. A1 represents task attempt 1 of A.
CompletedTasks contains A and B. A1 and A2 are succeeded. A2 was a rerun of A1.
B1 is succeeded and B2 was running when AM crashed.
A1- container request is processed. It uses the nodeid info from A1 to work.
B1- container request is processed. It uses the nodeid info from B1 to work.
A1- Succeeded. removes A1
B1- Succeeded. removes B1
A2- container request is processed. It uses the nodeid info from A2 to work
B2- container request is processed. It uses the nodeid info from B2 to work.
But there is no such info as it is populated on task attempt completion. AM
crashed here while trying to resolve the nodeid.
If AM had not crashed the following would have happened
A2- Succeeded. removes A2
There is no FAILED/KILLED/SUCCEEDED event for B2 since it was running when the
AM crashed. So it seems the AM would never move out of recovery.

If the above is correct, there seems to be 2 problems
1) While recovery is in process, event handling for task attempts that are not
in a completed state. I am not sure if the recovery design allows this and the
current crash is simply a case of missing info.
2) Expecting every task attempt of a completedTask to have a
KILLED/FAILED/SUCCEEDED entry. This seems to be clearly wrong in the current
scenario.

AM Recovery expects all attempts of a completed task to also be completed.
--

Key: MAPREDUCE-4128
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4128
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
Fix For: 3.0.0

The AM seems to assume that all attempts of a completed task (from a previous
AM incarnation) would also be completed. There is at least one case in which
this does not hold. Case being cancellation of a completed task resulting in
a new running attempt.

[jira] [Updated] (MAPREDUCE-3493) Add the default mapreduce.shuffle.port property to mapred-default.xml

2012-04-09 Thread madhukara phatak (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

madhukara phatak updated MAPREDUCE-3493:


Attachment: MAPREDUCE-3493-1.patch

Fixed core tests errors.

 Add the default mapreduce.shuffle.port property to mapred-default.xml
 -

 Key: MAPREDUCE-3493
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3493
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.0
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
Priority: Minor
  Labels: newbie
 Attachments: MAPREDUCE-3493-1.patch, MAPREDUCE-3493.patch


 I faced this issue when trying to run multiple Hadoop MR2 instances on the 
 same node. The default value for this property is hardcoded in the 
 ShuffleHandler.java class so it results in port conflicts. The issue is 
 resolved if you set the property value in your conf files. But the absence of 
 this property from *-default.xml files is confusing. So It'll be cleaner to 
 move this property to mapred-default.xml, so its default value can be easily 
 identified and changed if needed. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4099) ApplicationMaster may fail to remove staging directory


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-4099:
--

Status: Open  (was: Patch Available)

 ApplicationMaster may fail to remove staging directory
 --

 Key: MAPREDUCE-4099
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4099
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
 Attachments: MAPREDUCE-4099.patch


 When the ApplicationMaster shuts down it's supposed to remove the staging 
 directory, assuming properties weren't set to override this behavior. During 
 shutdown the AM tells the ResourceManager that it has finished before it 
 cleans up the staging directory.  However upon hearing the AM has finished, 
 the RM turns right around and kills the AM container.  If the AM is too slow, 
 the AM will be killed before the staging directory is removed.
 We're seeing the AM lose this race fairly consistently on our clusters, and 
 the lack of staging directory cleanup quickly leads to filesystem quota 
 issues for some users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4099) ApplicationMaster may fail to remove staging directory


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-4099:
--

Attachment: MAPREDUCE-4099.patch

Apparently I need to submit a new patch to convince Jenkins to run again.

 ApplicationMaster may fail to remove staging directory
 --

 Key: MAPREDUCE-4099
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4099
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
 Attachments: MAPREDUCE-4099.patch, MAPREDUCE-4099.patch


 When the ApplicationMaster shuts down it's supposed to remove the staging 
 directory, assuming properties weren't set to override this behavior. During 
 shutdown the AM tells the ResourceManager that it has finished before it 
 cleans up the staging directory.  However upon hearing the AM has finished, 
 the RM turns right around and kills the AM container.  If the AM is too slow, 
 the AM will be killed before the staging directory is removed.
 We're seeing the AM lose this race fairly consistently on our clusters, and 
 the lack of staging directory cleanup quickly leads to filesystem quota 
 issues for some users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4099) ApplicationMaster may fail to remove staging directory


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-4099:
--

Status: Patch Available  (was: Open)

 ApplicationMaster may fail to remove staging directory
 --

 Key: MAPREDUCE-4099
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4099
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
 Attachments: MAPREDUCE-4099.patch, MAPREDUCE-4099.patch


 When the ApplicationMaster shuts down it's supposed to remove the staging 
 directory, assuming properties weren't set to override this behavior. During 
 shutdown the AM tells the ResourceManager that it has finished before it 
 cleans up the staging directory.  However upon hearing the AM has finished, 
 the RM turns right around and kills the AM container.  If the AM is too slow, 
 the AM will be killed before the staging directory is removed.
 We're seeing the AM lose this race fairly consistently on our clusters, and 
 the lack of staging directory cleanup quickly leads to filesystem quota 
 issues for some users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4117) mapred job -status throws NullPointerException


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250054#comment-13250054
 ] 

Hadoop QA commented on MAPREDUCE-4117:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12521978/MAPREDUCE-4117-1.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 1 new or modified test 
files.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 eclipse:eclipse.  The patch built with eclipse:eclipse.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

-1 core tests.  The patch failed these unit tests:
  
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
  
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
  
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
  
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2175//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2175//console

This message is automatically generated.

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117-1.patch, MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4117) mapred job -status throws NullPointerException

2012-04-09 Thread Jonathan Eagles (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250061#comment-13250061
 ] 

Jonathan Eagles commented on MAPREDUCE-4117:


+1 lgtm
np. this will be a great addition to the cli. i have verified that the test 
fail without the patch and succeed with the patch.


 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117-1.patch, MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4117) mapred job -status throws NullPointerException

2012-04-09 Thread Jonathan Eagles (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250073#comment-13250073
 ] 

Jonathan Eagles commented on MAPREDUCE-4117:


I should have stated +1 (non-binding). A committer will need to put this in

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Attachments: MAPREDUCE-4117-1.patch, MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface

[
https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250075#comment-13250075
]

Thomas Graves commented on MAPREDUCE-4059:
--

For the most part looks good.

can you make the formatting in CachedHistoryStorage.java consistent - some
places are missing a space between if and (.

I think it would be more extensible to have the getPartialJobs interface take a
class rather then all the individual query parameters. If you want to push this
off to another Jira I'm fine with that since its marked as
InterfaceStability.Unstable.

The history server should have a separate pluggable storage/query interface
---

[jira] [Commented] (MAPREDUCE-4099) ApplicationMaster may fail to remove staging directory

[
https://issues.apache.org/jira/browse/MAPREDUCE-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250093#comment-13250093
]

Hadoop QA commented on MAPREDUCE-4099:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521985/MAPREDUCE-4099.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 7 new or modified test
files.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService

org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry

org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization

org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2177//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2177//console

This message is automatically generated.

ApplicationMaster may fail to remove staging directory
--

[jira] [Commented] (MAPREDUCE-3493) Add the default mapreduce.shuffle.port property to mapred-default.xml

[
https://issues.apache.org/jira/browse/MAPREDUCE-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250098#comment-13250098
]

Hadoop QA commented on MAPREDUCE-3493:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12521983/MAPREDUCE-3493-1.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2176//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2176//console

This message is automatically generated.

Add the default mapreduce.shuffle.port property to mapred-default.xml
-

Key: MAPREDUCE-3493
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3493
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
Priority: Minor
Labels: newbie
Attachments: MAPREDUCE-3493-1.patch, MAPREDUCE-3493.patch

I faced this issue when trying to run multiple Hadoop MR2 instances on the
same node. The default value for this property is hardcoded in the
ShuffleHandler.java class so it results in port conflicts. The issue is
resolved if you set the property value in your conf files. But the absence of
this property from *-default.xml files is confusing. So It'll be cleaner to
move this property to mapred-default.xml, so its default value can be easily
identified and changed if needed.

[jira] [Updated] (MAPREDUCE-4017) Add jobname to jobsummary log

2012-04-09 Thread Thomas Graves (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Graves updated MAPREDUCE-4017:
-

Target Version/s: 1.1.0, 0.23.3  (was: 0.23.3, 1.1.0)
  Status: Patch Available  (was: Open)

 Add jobname to jobsummary log
 -

 Key: MAPREDUCE-4017
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4017
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver, jobtracker
Affects Versions: 0.23.0, 0.20.205.0
Reporter: Koji Noguchi
Assignee: Thomas Graves
Priority: Trivial
 Attachments: MAPREDUCE-4017-v0.20-1.patch, MAPREDUCE-4017.patch


 We occasionally use jobsummary from the JobTracker to collect users' slot 
 usage on our clusters.  It would be useful if the jobname was part of this 
 jobsummary so that I don't need to join with other logs.
 Same jobsummary.
   2012-03-15 16:05:55,919 INFO mapred.JobInProgress$JobSummary: 
 jobId=job_201202160624_1089972,submitTime=1331827523632,launchTime=1331827528197,firstMapTaskLaunchTime=1331827536917,firstReduceTaskLaunchTime=1331827541251,firstJobSetupTaskLaunchTime=1331827528200,firstJobCleanupTaskLaunchTime=1331827551655,finishTime=1331827555919,numMaps=1,numSlotsPerMap=1,numReduces=1,numSlotsPerReduce=1,user=tortuga,queue=queue1,status=SUCCEEDED,mapSlotSeconds=13,reduceSlotsSeconds=10,clusterMapCapacity=,clusterReduceCapacity=
 I'd like to see jobName added to the end.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4017) Add jobname to jobsummary log

2012-04-09 Thread Thomas Graves (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Graves updated MAPREDUCE-4017:
-

Attachment: MAPREDUCE-4017.patch

Thanks for the patch Koji.  Here is the corresponding trunk/branch-2 patch.  

I've tested on both branch1 and trunk on single node clusters and verified the 
output as well.

 Add jobname to jobsummary log
 -

 Key: MAPREDUCE-4017
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4017
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver, jobtracker
Affects Versions: 0.20.205.0, 0.23.0
Reporter: Koji Noguchi
Assignee: Thomas Graves
Priority: Trivial
 Attachments: MAPREDUCE-4017-v0.20-1.patch, MAPREDUCE-4017.patch


 We occasionally use jobsummary from the JobTracker to collect users' slot 
 usage on our clusters.  It would be useful if the jobname was part of this 
 jobsummary so that I don't need to join with other logs.
 Same jobsummary.
   2012-03-15 16:05:55,919 INFO mapred.JobInProgress$JobSummary: 
 jobId=job_201202160624_1089972,submitTime=1331827523632,launchTime=1331827528197,firstMapTaskLaunchTime=1331827536917,firstReduceTaskLaunchTime=1331827541251,firstJobSetupTaskLaunchTime=1331827528200,firstJobCleanupTaskLaunchTime=1331827551655,finishTime=1331827555919,numMaps=1,numSlotsPerMap=1,numReduces=1,numSlotsPerReduce=1,user=tortuga,queue=queue1,status=SUCCEEDED,mapSlotSeconds=13,reduceSlotsSeconds=10,clusterMapCapacity=,clusterReduceCapacity=
 I'd like to see jobName added to the end.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4017) Add jobname to jobsummary log

[
https://issues.apache.org/jira/browse/MAPREDUCE-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250118#comment-13250118
]

Hadoop QA commented on MAPREDUCE-4017:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522004/MAPREDUCE-4017.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService

org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry

org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization

org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2178//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2178//console

This message is automatically generated.

Add jobname to jobsummary log
-

Key: MAPREDUCE-4017
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4017
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver, jobtracker
Affects Versions: 0.20.205.0, 0.23.0
Reporter: Koji Noguchi
Assignee: Thomas Graves
Priority: Trivial
Attachments: MAPREDUCE-4017-v0.20-1.patch, MAPREDUCE-4017.patch

We occasionally use jobsummary from the JobTracker to collect users' slot
usage on our clusters. It would be useful if the jobname was part of this
jobsummary so that I don't need to join with other logs.
Same jobsummary.
2012-03-15 16:05:55,919 INFO mapred.JobInProgress$JobSummary:
jobId=job_201202160624_1089972,submitTime=1331827523632,launchTime=1331827528197,firstMapTaskLaunchTime=1331827536917,firstReduceTaskLaunchTime=1331827541251,firstJobSetupTaskLaunchTime=1331827528200,firstJobCleanupTaskLaunchTime=1331827551655,finishTime=1331827555919,numMaps=1,numSlotsPerMap=1,numReduces=1,numSlotsPerReduce=1,user=tortuga,queue=queue1,status=SUCCEEDED,mapSlotSeconds=13,reduceSlotsSeconds=10,clusterMapCapacity=,clusterReduceCapacity=
I'd like to see jobName added to the end.

[jira] [Commented] (MAPREDUCE-3493) Add the default mapreduce.shuffle.port property to mapred-default.xml

2012-04-09 Thread Harsh J (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250119#comment-13250119
]

Harsh J commented on MAPREDUCE-3493:

Looks good. Can you also elaborate a bit more on what the ShuffleHandler is, in
the description field? May come handy to some.

Add the default mapreduce.shuffle.port property to mapred-default.xml
-

[jira] [Resolved] (MAPREDUCE-1233) Incorrect Waiting maps/reduces in Jobtracker metrics

2012-04-09 Thread Thomas Graves (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Graves resolved MAPREDUCE-1233.
--

   Resolution: Fixed
Fix Version/s: 0.20.203.0

 Incorrect Waiting maps/reduces in Jobtracker metrics 
 -

 Key: MAPREDUCE-1233
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1233
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: V.Karthikeyan
Assignee: Luke Lu
 Fix For: 0.20.203.0

 Attachments: mr-1233-y20s-v1.patch, mr-1233-y20s-v2.patch


 Waiting Maps/Reduces are incorrect in Jobtracker metrics when a job fails. 
 when a map/reduce fails(during job failure), waiting maps/reduce got 
 incremented and doesn't get decremented even after job cleanup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4017) Add jobname to jobsummary log


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250128#comment-13250128
 ] 

Thomas Graves commented on MAPREDUCE-4017:
--

test failures are known and not caused by this change.

 Add jobname to jobsummary log
 -

 Key: MAPREDUCE-4017
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4017
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver, jobtracker
Affects Versions: 0.20.205.0, 0.23.0
Reporter: Koji Noguchi
Assignee: Thomas Graves
Priority: Trivial
 Attachments: MAPREDUCE-4017-v0.20-1.patch, MAPREDUCE-4017.patch


 We occasionally use jobsummary from the JobTracker to collect users' slot 
 usage on our clusters.  It would be useful if the jobname was part of this 
 jobsummary so that I don't need to join with other logs.
 Same jobsummary.
   2012-03-15 16:05:55,919 INFO mapred.JobInProgress$JobSummary: 
 jobId=job_201202160624_1089972,submitTime=1331827523632,launchTime=1331827528197,firstMapTaskLaunchTime=1331827536917,firstReduceTaskLaunchTime=1331827541251,firstJobSetupTaskLaunchTime=1331827528200,firstJobCleanupTaskLaunchTime=1331827551655,finishTime=1331827555919,numMaps=1,numSlotsPerMap=1,numReduces=1,numSlotsPerReduce=1,user=tortuga,queue=queue1,status=SUCCEEDED,mapSlotSeconds=13,reduceSlotsSeconds=10,clusterMapCapacity=,clusterReduceCapacity=
 I'd like to see jobName added to the end.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1233) Incorrect Waiting maps/reduces in Jobtracker metrics


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250127#comment-13250127
 ] 

Thomas Graves commented on MAPREDUCE-1233:
--

Yes, Koji is right, this patch was integrated into branch-1/20s in revision  
1077665 
(branches/branch-1.0/src/mapred/org/apache/hadoop/mapred/JobInProgress.java) - 
which according to changes.txt would have went into 0.20.202.0.

Moving this to resolved and can fix issue under MAPREDUCE-1238.

 Incorrect Waiting maps/reduces in Jobtracker metrics 
 -

 Key: MAPREDUCE-1233
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1233
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: V.Karthikeyan
Assignee: Luke Lu
 Fix For: 0.20.203.0

 Attachments: mr-1233-y20s-v1.patch, mr-1233-y20s-v2.patch


 Waiting Maps/Reduces are incorrect in Jobtracker metrics when a job fails. 
 when a map/reduce fails(during job failure), waiting maps/reduce got 
 incremented and doesn't get decremented even after job cleanup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-1238) mapred metrics shows negative count of waiting maps and reduces

2012-04-09 Thread Thomas Graves (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Graves reassigned MAPREDUCE-1238:


Assignee: Thomas Graves

 mapred metrics shows negative count of waiting maps and reduces 
 

 Key: MAPREDUCE-1238
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1238
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Ramya Sunil
Assignee: Thomas Graves
 Attachments: MAPREDUCE-1238-v0.20-1.patch


 Negative waiting_maps and waiting_reduces count is observed in the mapred 
 metrics

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4099) ApplicationMaster may fail to remove staging directory

2012-04-09 Thread Jason Lowe (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250151#comment-13250151
]

Jason Lowe commented on MAPREDUCE-4099:
---

All of the reported test failures appear to be unrelated to the patch. They
all fail because a ResourceManager process can't start due to a socket bind
problem -- a runaway RM process on the build machine, perhaps? I ran the RM
unit tests locally with this patch and they all pass.

I also manually tested the patch with a single-node cluster running sleep and
wordcount jobs. Also connected the debugger to the ApplicationMaster, causing
it to linger artificially in the FINISHING state to verify killing or expiring
an application in the FINISHING state behaves properly.

ApplicationMaster may fail to remove staging directory
--

[jira] [Commented] (MAPREDUCE-4128) AM Recovery expects all attempts of a completed task to also be completed.


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250185#comment-13250185
 ] 

Bikas Saha commented on MAPREDUCE-4128:
---

I have managed to repro this using a modified TestFetchFailure. So this is 
unrelated to the specific changes in MAPREDUCE-3921.

 AM Recovery expects all attempts of a completed task to also be completed.
 --

 Key: MAPREDUCE-4128
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4128
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
 Fix For: 3.0.0


 The AM seems to assume that all attempts of a completed task (from a previous 
 AM incarnation) would also be completed. There is at least one case in which 
 this does not hold. Case being cancellation of a completed task resulting in 
 a new running attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1238) mapred metrics shows negative count of waiting maps and reduces


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250207#comment-13250207
 ] 

Thomas Graves commented on MAPREDUCE-1238:
--

I believe this will still miss a case when it is killed while in progress of 
initializing the job.  If it receives a kill during that initialization it 
waits until init is done and then throws to do the kill.  The tasksInited will 
not have been set to true because the throw happens right before that is set 
and thus the metrics won't be properly decremented.  Working on a fix for that.

 mapred metrics shows negative count of waiting maps and reduces 
 

 Key: MAPREDUCE-1238
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1238
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Ramya Sunil
Assignee: Thomas Graves
 Attachments: MAPREDUCE-1238-v0.20-1.patch


 Negative waiting_maps and waiting_reduces count is observed in the mapred 
 metrics

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4117) mapred job -status throws NullPointerException

2012-04-09 Thread Robert Joseph Evans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4117:
---

   Resolution: Fixed
Fix Version/s: 2.0.0
   0.23.3
   Status: Resolved  (was: Patch Available)

I agree with Jon, +1.  I merged this into trunk, branch-2, and branch-0.23

 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Fix For: 0.23.3, 2.0.0

 Attachments: MAPREDUCE-4117-1.patch, MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4117) mapred job -status throws NullPointerException


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250228#comment-13250228
 ] 

Hudson commented on MAPREDUCE-4117:
---

Integrated in Hadoop-Common-trunk-Commit #2031 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2031/])
MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K 
via bobby) (Revision 1311479)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311479
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java


 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Fix For: 0.23.3, 2.0.0

 Attachments: MAPREDUCE-4117-1.patch, MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4117) mapred job -status throws NullPointerException


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250242#comment-13250242
 ] 

Hudson commented on MAPREDUCE-4117:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #2042 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2042/])
MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K 
via bobby) (Revision 1311479)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311479
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java


 mapred job -status throws NullPointerException
 --

 Key: MAPREDUCE-4117
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4117
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Fix For: 0.23.3, 2.0.0

 Attachments: MAPREDUCE-4117-1.patch, MAPREDUCE-4117.patch


 {code:xml}
 dev@ubuntudev-linux:~/hadoop/hadoop-trunk/bin$ ./mapred job -status 
 job_1333408894669_0001
 Exception in thread main java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.Job.getTaskFailureEventString(Job.java:512)
 at org.apache.hadoop.mapreduce.Job.toString(Job.java:463)
 at java.lang.String.valueOf(String.java:2838)
 at java.io.PrintStream.println(PrintStream.java:788)
 at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:255)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
 at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1244)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

2012-04-09 Thread Mariappan Asokan (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250275#comment-13250275
]

Mariappan Asokan commented on MAPREDUCE-4049:
-

Hi Avner,
I worked on MAPREDUCE-2454(to make sort pluggable in Hadoop) and posted a
patch on top of trunk version 1221902 a while back. The patch was created on
top of the trunk since ReduceTask.java was already refactored nicely and I was
advised to work on the trunk version.

Please take a look at the patch file mapreduce-2454.patch posted in
MAPREDUCE-2454. If you want, I can post a patch on top of the latest trunk.

The patch decoupled the merge from shuffle by creating ShuffleRunner and
ShuffleCallback interfaces. The MergeManager implements the ShuffleCallback
and the shuffle itself implements ShuffleRunner interface.

Since you are making shuffle as pluggable, I notice some overlapping changes.
If I can be of any assistance to reduce the conflict between our patches,
please let me know. Meanwhile, I will go over the details of your patch and
get back. Do you have a patch created on top of trunk?

Also, I would like to hear opinions from other developers who have shown
interest in this Jira.

plugin for generic shuffle service
--

Key: MAPREDUCE-4049
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: performance, task, tasktracker
Affects Versions: 0.23.1, 1.0.1
Reporter: Avner BenHanoch
Labels: merge, plugin, rdma, shuffle
Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle
Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf,
MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz,
test.diff

Support generic shuffle service as set of two plugins: ShuffleProvider
ShuffleConsumer.
This will satisfy the following needs:
# Better shuffle and merge performance. For example: we are working on
shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE,
or Infiniband) instead of using the current HTTP shuffle. Based on the fast
RDMA shuffle, the plugin can also utilize a suitable merge approach during
the intermediate merges. Hence, getting much better performance.
# Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden
dependency of NodeManager with a specific version of mapreduce shuffle
(currently targeted to 0.24.0).
References:
# Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu
from Auburn University with others,
[http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
# I am attaching 2 documents with suggested Top Level Design for both plugins
(currently, based on 1.0 branch)

[jira] [Updated] (MAPREDUCE-4105) Yarn RackResolver ignores rack configurations

2012-04-09 Thread Tom White (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom White updated MAPREDUCE-4105:
-

   Resolution: Fixed
Fix Version/s: (was: 0.23.2)
   2.0.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

I've just committed this. Thanks Ahmed!

 Yarn RackResolver ignores rack configurations
 -

 Key: MAPREDUCE-4105
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4105
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4105.patch, MAPREDUCE-4105_rev2.patch, 
 MAPREDUCE-4105_rev3.patch


 Incorrect mappings because the Yarn RackResolver ignores rack configurations. 
 This can be verified by inspecting the resource manager web ui that lists all 
 the nodes, all of them show up with /default-rack regardless of the output 
 from the script specified using net.topology.script.file.name configuration 
 property.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4129) Lots of unneeded counters log messages

2012-04-09 Thread Ahmed Radwan (Created) (JIRA)

Lots of unneeded counters log messages
--

 Key: MAPREDUCE-4129
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4129
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Fix For: 0.23.2


Huge number of the same WARN messages are written. We only need to write each 
distinct message once. The messages are of the form:

{code}
2012-04-05 03:55:04,166 WARN mapreduce.Counters: Group {oldGroup} is 
deprecated. Use {newGroup} instead
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4129) Lots of unneeded counters log messages

2012-04-09 Thread Ahmed Radwan (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-4129:


Attachment: MAPREDUCE-4129.patch

Attaching a patch addressing this issue. The patch modifies the behavior to 
only write the log message the first time the oldGroupName is found, instead of 
every time getGroup(..) is called.
I don't think we need a test case for this change as it is a straightforward 
change in message logging.

 Lots of unneeded counters log messages
 --

 Key: MAPREDUCE-4129
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4129
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Fix For: 0.23.2

 Attachments: MAPREDUCE-4129.patch


 Huge number of the same WARN messages are written. We only need to write each 
 distinct message once. The messages are of the form:
 {code}
 2012-04-05 03:55:04,166 WARN mapreduce.Counters: Group {oldGroup} is 
 deprecated. Use {newGroup} instead
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4129) Lots of unneeded counters log messages

2012-04-09 Thread Ahmed Radwan (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-4129:


Status: Patch Available  (was: Open)

 Lots of unneeded counters log messages
 --

 Key: MAPREDUCE-4129
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4129
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Fix For: 0.23.2

 Attachments: MAPREDUCE-4129.patch


 Huge number of the same WARN messages are written. We only need to write each 
 distinct message once. The messages are of the form:
 {code}
 2012-04-05 03:55:04,166 WARN mapreduce.Counters: Group {oldGroup} is 
 deprecated. Use {newGroup} instead
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3409) Incorrect custom task status when running on MR2

2012-04-09 Thread Tom White (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom White updated MAPREDUCE-3409:
-

Target Version/s: 2.0.0

 Incorrect custom task status when running on MR2
 

 Key: MAPREDUCE-3409
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3409
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.0
Reporter: Ahmed Radwan

 To reproduce this problem:
 1- In your mapper setup() set: 
 {code}
 context.setStatus(myStatus)
 {code}
 2- When the job finishes:
 {code} 
 TaskReport[] reports = job.getTaskReports(TaskType.MAP);
 assertEquals(myStatus, reports[0].getState());
 {code}
 The returned status from reports[0].getState() is SUCCEEDED as opposed to 
 the expected myStatus value.
 This exact code work fine on MR1. I saw this issue when tried running the 
 TestTaskContext test cases on MR2.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4105) Yarn RackResolver ignores rack configurations


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250320#comment-13250320
 ] 

Hudson commented on MAPREDUCE-4105:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #2043 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2043/])
MAPREDUCE-4105. Yarn RackResolver ignores rack configurations. Contributed 
by Ahmed Radwan. (Revision 1311520)

 Result = SUCCESS
tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311520
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolverScriptBasedMapping.java


 Yarn RackResolver ignores rack configurations
 -

 Key: MAPREDUCE-4105
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4105
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4105.patch, MAPREDUCE-4105_rev2.patch, 
 MAPREDUCE-4105_rev3.patch


 Incorrect mappings because the Yarn RackResolver ignores rack configurations. 
 This can be verified by inspecting the resource manager web ui that lists all 
 the nodes, all of them show up with /default-rack regardless of the output 
 from the script specified using net.topology.script.file.name configuration 
 property.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3869) Distributed shell application fails with NoClassDefFoundError

2012-04-09 Thread Siddharth Seth (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250322#comment-13250322
 ] 

Siddharth Seth commented on MAPREDUCE-3869:
---

+1. Will commit this shortly.

 Distributed shell application fails with NoClassDefFoundError
 -

 Key: MAPREDUCE-3869
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3869
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Blocker
 Fix For: 0.23.2

 Attachments: MAPREDUCE-3869-1.patch, MAPREDUCE-3869-2.patch, 
 MAPREDUCE-3869.patch


 Distributed shell application always fails to start the application master 
 with the following error.
 \\
 {code:xml}
 12/02/16 05:35:25 FATAL distributedshell.ApplicationMaster: Error running 
 ApplicationMaster
 java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.init(ApplicationMaster.java:252)
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:195)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.yarn.ipc.YarnRPC
   at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
   at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
   ... 2 more
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3869) Distributed shell application fails with NoClassDefFoundError

2012-04-09 Thread Siddharth Seth (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated MAPREDUCE-3869:
--

   Resolution: Fixed
Fix Version/s: (was: 0.23.2)
   2.0.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to trunk and branch-2. Thanks Devaraj.

 Distributed shell application fails with NoClassDefFoundError
 -

 Key: MAPREDUCE-3869
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3869
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Blocker
 Fix For: 2.0.0

 Attachments: MAPREDUCE-3869-1.patch, MAPREDUCE-3869-2.patch, 
 MAPREDUCE-3869.patch


 Distributed shell application always fails to start the application master 
 with the following error.
 \\
 {code:xml}
 12/02/16 05:35:25 FATAL distributedshell.ApplicationMaster: Error running 
 ApplicationMaster
 java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.init(ApplicationMaster.java:252)
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:195)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.yarn.ipc.YarnRPC
   at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
   at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
   ... 2 more
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4105) Yarn RackResolver ignores rack configurations


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250328#comment-13250328
 ] 

Hudson commented on MAPREDUCE-4105:
---

Integrated in Hadoop-Common-trunk-Commit #2032 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2032/])
MAPREDUCE-4105. Yarn RackResolver ignores rack configurations. Contributed 
by Ahmed Radwan. (Revision 1311520)

 Result = SUCCESS
tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311520
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolverScriptBasedMapping.java


 Yarn RackResolver ignores rack configurations
 -

 Key: MAPREDUCE-4105
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4105
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4105.patch, MAPREDUCE-4105_rev2.patch, 
 MAPREDUCE-4105_rev3.patch


 Incorrect mappings because the Yarn RackResolver ignores rack configurations. 
 This can be verified by inspecting the resource manager web ui that lists all 
 the nodes, all of them show up with /default-rack regardless of the output 
 from the script specified using net.topology.script.file.name configuration 
 property.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3869) Distributed shell application fails with NoClassDefFoundError


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250329#comment-13250329
 ] 

Hudson commented on MAPREDUCE-3869:
---

Integrated in Hadoop-Common-trunk-Commit #2032 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2032/])
MAPREDUCE-3869. Fix classpath for DistributedShell application. 
(Contributed by Devaraj K) (Revision 1311523)

 Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311523
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java


 Distributed shell application fails with NoClassDefFoundError
 -

 Key: MAPREDUCE-3869
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3869
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Blocker
 Fix For: 2.0.0

 Attachments: MAPREDUCE-3869-1.patch, MAPREDUCE-3869-2.patch, 
 MAPREDUCE-3869.patch


 Distributed shell application always fails to start the application master 
 with the following error.
 \\
 {code:xml}
 12/02/16 05:35:25 FATAL distributedshell.ApplicationMaster: Error running 
 ApplicationMaster
 java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.init(ApplicationMaster.java:252)
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:195)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.yarn.ipc.YarnRPC
   at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
   at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
   ... 2 more
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3869) Distributed shell application fails with NoClassDefFoundError


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250334#comment-13250334
 ] 

Hudson commented on MAPREDUCE-3869:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2107 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2107/])
MAPREDUCE-3869. Fix classpath for DistributedShell application. 
(Contributed by Devaraj K) (Revision 1311523)

 Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311523
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java


 Distributed shell application fails with NoClassDefFoundError
 -

 Key: MAPREDUCE-3869
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3869
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Blocker
 Fix For: 2.0.0

 Attachments: MAPREDUCE-3869-1.patch, MAPREDUCE-3869-2.patch, 
 MAPREDUCE-3869.patch


 Distributed shell application always fails to start the application master 
 with the following error.
 \\
 {code:xml}
 12/02/16 05:35:25 FATAL distributedshell.ApplicationMaster: Error running 
 ApplicationMaster
 java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.init(ApplicationMaster.java:252)
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:195)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.yarn.ipc.YarnRPC
   at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
   at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
   ... 2 more
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4105) Yarn RackResolver ignores rack configurations


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250333#comment-13250333
 ] 

Hudson commented on MAPREDUCE-4105:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2107 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2107/])
MAPREDUCE-4105. Yarn RackResolver ignores rack configurations. Contributed 
by Ahmed Radwan. (Revision 1311520)

 Result = SUCCESS
tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311520
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolverScriptBasedMapping.java


 Yarn RackResolver ignores rack configurations
 -

 Key: MAPREDUCE-4105
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4105
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4105.patch, MAPREDUCE-4105_rev2.patch, 
 MAPREDUCE-4105_rev3.patch


 Incorrect mappings because the Yarn RackResolver ignores rack configurations. 
 This can be verified by inspecting the resource manager web ui that lists all 
 the nodes, all of them show up with /default-rack regardless of the output 
 from the script specified using net.topology.script.file.name configuration 
 property.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4129) Lots of unneeded counters log messages

2012-04-09 Thread Tsz Wo (Nicholas), SZE (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250338#comment-13250338
]

Hadoop QA commented on MAPREDUCE-4129:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522044/MAPREDUCE-4129.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService

org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry

org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization

org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2179//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2179//console

This message is automatically generated.

Lots of unneeded counters log messages
--

Key: MAPREDUCE-4129
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4129
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
Fix For: 0.23.2

Attachments: MAPREDUCE-4129.patch

Huge number of the same WARN messages are written. We only need to write each
distinct message once. The messages are of the form:
{code}
2012-04-05 03:55:04,166 WARN mapreduce.Counters: Group {oldGroup} is
deprecated. Use {newGroup} instead
{code}

[jira] [Updated] (MAPREDUCE-4128) AM Recovery expects all attempts of a completed task to also be completed.

2012-04-09 Thread Bikas Saha (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated MAPREDUCE-4128:
--

Attachment: MAPREDUCE-4128.patch

 AM Recovery expects all attempts of a completed task to also be completed.
 --

 Key: MAPREDUCE-4128
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4128
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
 Fix For: 3.0.0

 Attachments: MAPREDUCE-4128.patch


 The AM seems to assume that all attempts of a completed task (from a previous 
 AM incarnation) would also be completed. There is at least one case in which 
 this does not hold. Case being cancellation of a completed task resulting in 
 a new running attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4057) Compilation error in RAID


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250352#comment-13250352
 ] 

Tsz Wo (Nicholas), SZE commented on MAPREDUCE-4057:
---

Raid still cannot be compiled after the patch:
{noformat}
compile-test:
 [echo] contrib: raid
[javac] 
/Users/szetszwo/hadoop/t-commit/hadoop-mapreduce-project/src/contrib/build-contrib.xml:227:
 warning: 'includeantruntime' was not set, defaulting to 
build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 20 source files to 
/Users/szetszwo/hadoop/t-commit/hadoop-mapreduce-project/build/contrib/raid/test
[javac] 
/Users/szetszwo/hadoop/t-commit/hadoop-mapreduce-project/src/contrib/raid/src/test/org/apache/hadoop/raid/TestBlockFixer.java:42:
 cannot find symbol
[javac] symbol  : class MiniMRCluster
[javac] location: package org.apache.hadoop.mapred
[javac] import org.apache.hadoop.mapred.MiniMRCluster;
[javac]^
...
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 12 errors
{noformat}

 Compilation error in RAID 
 --

 Key: MAPREDUCE-4057
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4057
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/raid
Reporter: Tsz Wo (Nicholas), SZE
 Attachments: MAPREDUCE-4057.patch


 {noformat}
 [javac] Compiling 33 source files to 
 /Users/szetszwo/hadoop/t2/hadoop-mapreduce-project/build/contrib/raid/classes
 [javac] 
 /Users/szetszwo/hadoop/t2/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java:42:
  unreported exception org.apache.hadoop.ipc.StandbyException; must be caught 
 or declared to be thrown
 [javac] return namesystem.getFileInfo(src, resolveLink);
 [javac]  ^
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4128) AM Recovery expects all attempts of a completed task to also be completed.

[
https://issues.apache.org/jira/browse/MAPREDUCE-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250355#comment-13250355
]

Bikas Saha commented on MAPREDUCE-4128:
---

The current recovery mechanism seems to be designed to recover completed tasks.
Hence it assumes that all attempts of such tasks would also be complete. So it
loads completed tasks and replays them till the replay completes all attempts
of all completed tasks.
This will break whenever there is an attempt running after a previously
successful completion because the replay will not have info to correctly work
on running attempts.
Scenario 1 : MAPREDUCE-3921 introduces such an instance because it re-runs
successful map tasks if the successful attempts had run on a bad machine.
Scenario 2 : Even in the current code, when a successful map is rerun because
of too many fetch failures, the above scenario is produced and caused a failure
in recovery.
The proposed solution in the patch is to make sure that if a task is re-run
then it is not marked as completed during recovery. The JobHistoryParser has
been changed to remove the SUCCEEDED status on a task if the successful
attempt of that task later reports a failure. This fixes the repro case
mentioned above. I have improved that testcase to cover Scenario 1. Scenario 2
will be covered in MAPREDUCE-3921.
I am expecting the patch to introduce 3 additional warnings because of raw
types in event handling (similar to existing warnings).
I compiled a broken rumen class and test assuming the new field added to
TaskFinishedEvent is not relevant to them.

AM Recovery expects all attempts of a completed task to also be completed.
--

Attachments: MAPREDUCE-4128.patch

[jira] [Commented] (MAPREDUCE-3869) Distributed shell application fails with NoClassDefFoundError

2012-04-09 Thread Tsz Wo (Nicholas), SZE (Updated) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250356#comment-13250356
 ] 

Hudson commented on MAPREDUCE-3869:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #2044 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2044/])
MAPREDUCE-3869. Fix classpath for DistributedShell application. 
(Contributed by Devaraj K) (Revision 1311523)

 Result = ABORTED
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311523
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java


 Distributed shell application fails with NoClassDefFoundError
 -

 Key: MAPREDUCE-3869
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3869
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Blocker
 Fix For: 2.0.0

 Attachments: MAPREDUCE-3869-1.patch, MAPREDUCE-3869-2.patch, 
 MAPREDUCE-3869.patch


 Distributed shell application always fails to start the application master 
 with the following error.
 \\
 {code:xml}
 12/02/16 05:35:25 FATAL distributedshell.ApplicationMaster: Error running 
 ApplicationMaster
 java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.init(ApplicationMaster.java:252)
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:195)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.yarn.ipc.YarnRPC
   at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
   at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
   ... 2 more
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4057) Compilation error in RAID


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated MAPREDUCE-4057:
--

Assignee: Devaraj K

Devaraj, thanks for posting a patch.  Could you also fix the errors in the 
tests?

 Compilation error in RAID 
 --

 Key: MAPREDUCE-4057
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4057
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/raid
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Devaraj K
 Attachments: MAPREDUCE-4057.patch


 {noformat}
 [javac] Compiling 33 source files to 
 /Users/szetszwo/hadoop/t2/hadoop-mapreduce-project/build/contrib/raid/classes
 [javac] 
 /Users/szetszwo/hadoop/t2/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java:42:
  unreported exception org.apache.hadoop.ipc.StandbyException; must be caught 
 or declared to be thrown
 [javac] return namesystem.getFileInfo(src, resolveLink);
 [javac]  ^
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4128) AM Recovery expects all attempts of a completed task to also be completed.

2012-04-09 Thread Bikas Saha (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated MAPREDUCE-4128:
--

Status: Patch Available  (was: Open)

 AM Recovery expects all attempts of a completed task to also be completed.
 --

 Key: MAPREDUCE-4128
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4128
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
 Fix For: 3.0.0

 Attachments: MAPREDUCE-4128.patch


 The AM seems to assume that all attempts of a completed task (from a previous 
 AM incarnation) would also be completed. There is at least one case in which 
 this does not hold. Case being cancellation of a completed task resulting in 
 a new running attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4128) AM Recovery expects all attempts of a completed task to also be completed.


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250360#comment-13250360
 ] 

Bikas Saha commented on MAPREDUCE-4128:
---

Correction.
bq.This fixes the repro case mentioned above. I have improved that testcase to 
cover -Scenario 1- Scenario 2. -Scenario 2- Scenario 1 will be covered in 
MAPREDUCE-3921.

 AM Recovery expects all attempts of a completed task to also be completed.
 --

 Key: MAPREDUCE-4128
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4128
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
 Fix For: 3.0.0

 Attachments: MAPREDUCE-4128.patch


 The AM seems to assume that all attempts of a completed task (from a previous 
 AM incarnation) would also be completed. There is at least one case in which 
 this does not hold. Case being cancellation of a completed task resulting in 
 a new running attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4128) AM Recovery expects all attempts of a completed task to also be completed.