[jira] [Updated] (MAPREDUCE-4906) testJobTrackerRestartsWithMissingJobFile failed in branch-1

[meng gong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

meng gong updated MAPREDUCE-4906:
-

Description: 

{noformat}
Testcase: testJobTrackerRestartsWithMissingJobFile took 164.41 sec
Caused an ERROR
test timed out after 12 milliseconds
java.lang.Exception: test timed out after 12 milliseconds

Testcase: testJobResubmission took 89.27 sec
Testcase: testJobTrackerRestartWithBadJobs took 122.568 sec
Testcase: testRestartCount took 10.896 sec
Testcase: testJobTrackerInfoCreation took 9.954 sec

{noformat}

> testJobTrackerRestartsWithMissingJobFile failed in branch-1
> ---
>
> Key: MAPREDUCE-4906
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4906
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 1.2.0
>Reporter: meng gong
>Assignee: meng gong
> Fix For: 1.2.0
>
>
> {noformat}
> Testcase: testJobTrackerRestartsWithMissingJobFile took 164.41 sec
> Caused an ERROR
> test timed out after 12 milliseconds
> java.lang.Exception: test timed out after 12 milliseconds
> Testcase: testJobResubmission took 89.27 sec
> Testcase: testJobTrackerRestartWithBadJobs took 122.568 sec
> Testcase: testRestartCount took 10.896 sec
> Testcase: testJobTrackerInfoCreation took 9.954 sec
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4906) testJobTrackerRestartsWithMissingJobFile failed in branch-1

[meng gong (JIRA)]

meng gong created MAPREDUCE-4906:


 Summary: testJobTrackerRestartsWithMissingJobFile failed in 
branch-1
 Key: MAPREDUCE-4906
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4906
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: 1.2.0
Reporter: meng gong
Assignee: meng gong
 Fix For: 1.2.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4906) testJobTrackerRestartsWithMissingJobFile failed in branch-1

[meng gong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

meng gong updated MAPREDUCE-4906:
-

Description: 
testJobTrackerRestartsWithMissingJobFile failed in branch-1 for timeout

{noformat}
Testcase: testJobTrackerRestartsWithMissingJobFile took 164.41 sec
Caused an ERROR
test timed out after 12 milliseconds
java.lang.Exception: test timed out after 12 milliseconds

Testcase: testJobResubmission took 89.27 sec
Testcase: testJobTrackerRestartWithBadJobs took 122.568 sec
Testcase: testRestartCount took 10.896 sec
Testcase: testJobTrackerInfoCreation took 9.954 sec

{noformat}

  was:

{noformat}
Testcase: testJobTrackerRestartsWithMissingJobFile took 164.41 sec
Caused an ERROR
test timed out after 12 milliseconds
java.lang.Exception: test timed out after 12 milliseconds

Testcase: testJobResubmission took 89.27 sec
Testcase: testJobTrackerRestartWithBadJobs took 122.568 sec
Testcase: testRestartCount took 10.896 sec
Testcase: testJobTrackerInfoCreation took 9.954 sec

{noformat}


> testJobTrackerRestartsWithMissingJobFile failed in branch-1
> ---
>
> Key: MAPREDUCE-4906
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4906
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 1.2.0
>Reporter: meng gong
>Assignee: meng gong
> Fix For: 1.2.0
>
>
> testJobTrackerRestartsWithMissingJobFile failed in branch-1 for timeout
> {noformat}
> Testcase: testJobTrackerRestartsWithMissingJobFile took 164.41 sec
> Caused an ERROR
> test timed out after 12 milliseconds
> java.lang.Exception: test timed out after 12 milliseconds
> Testcase: testJobResubmission took 89.27 sec
> Testcase: testJobTrackerRestartWithBadJobs took 122.568 sec
> Testcase: testRestartCount took 10.896 sec
> Testcase: testJobTrackerInfoCreation took 9.954 sec
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4905) test org.apache.hadoop.mapred.pipes

[Aleksey Gorshkov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Gorshkov updated MAPREDUCE-4905:


Attachment: MAPREDUCE-4905-trunk.patch

> test org.apache.hadoop.mapred.pipes
> ---
>
> Key: MAPREDUCE-4905
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4905
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
>Reporter: Aleksey Gorshkov
> Fix For: 3.0.0, 2.0.3-alpha, 0.23.6
>
> Attachments: MAPREDUCE-4905-trunk.patch
>
>
> tests for  org.apache.hadoop.mapred.pipes

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4905) test org.apache.hadoop.mapred.pipes

[Aleksey Gorshkov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Gorshkov updated MAPREDUCE-4905:


Description: 
tests for  org.apache.hadoop.mapred.pipes
patch MAPREDUCE-4905-trunk.patch for trunk, branch-2, branch-0.23

  was:tests for  org.apache.hadoop.mapred.pipes


> test org.apache.hadoop.mapred.pipes
> ---
>
> Key: MAPREDUCE-4905
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4905
> Project: Hadoop Map/Reduce
>  Issue Type: Test
>Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
>Reporter: Aleksey Gorshkov
> Fix For: 3.0.0, 2.0.3-alpha, 0.23.6
>
> Attachments: MAPREDUCE-4905-trunk.patch
>
>
> tests for  org.apache.hadoop.mapred.pipes
> patch MAPREDUCE-4905-trunk.patch for trunk, branch-2, branch-0.23

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4905) test org.apache.hadoop.mapred.pipes

[Aleksey Gorshkov (JIRA)]

Aleksey Gorshkov created MAPREDUCE-4905:
---

 Summary: test org.apache.hadoop.mapred.pipes
 Key: MAPREDUCE-4905
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4905
 Project: Hadoop Map/Reduce
  Issue Type: Test
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Aleksey Gorshkov
 Fix For: 3.0.0, 2.0.3-alpha, 0.23.6


tests for  org.apache.hadoop.mapred.pipes

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4904) TestMultipleLevelCaching failed in barnch-1

[meng gong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

meng gong updated MAPREDUCE-4904:
-

Description: 
TestMultipleLevelCaching will failed:
{noformat}
Testcase: testMultiLevelCaching took 30.406 sec
FAILED
Number of local maps expected:<0> but was:<1>
junit.framework.AssertionFailedError: Number of local maps expected:<0> but 
was:<1>
at 
org.apache.hadoop.mapred.TestRackAwareTaskPlacement.launchJobAndTestCounters(TestRackAwareTaskPlacement.java:78)
at 
org.apache.hadoop.mapred.TestMultipleLevelCaching.testCachingAtLevel(TestMultipleLevelCaching.java:113)
at 
org.apache.hadoop.mapred.TestMultipleLevelCaching.testMultiLevelCaching(TestMultipleLevelCaching.java:69)

{noformat}

> TestMultipleLevelCaching failed in barnch-1
> ---
>
> Key: MAPREDUCE-4904
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4904
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 1.2.0
>Reporter: meng gong
>Assignee: meng gong
> Fix For: 1.2.0
>
>
> TestMultipleLevelCaching will failed:
> {noformat}
> Testcase: testMultiLevelCaching took 30.406 sec
> FAILED
> Number of local maps expected:<0> but was:<1>
> junit.framework.AssertionFailedError: Number of local maps expected:<0> but 
> was:<1>
> at 
> org.apache.hadoop.mapred.TestRackAwareTaskPlacement.launchJobAndTestCounters(TestRackAwareTaskPlacement.java:78)
> at 
> org.apache.hadoop.mapred.TestMultipleLevelCaching.testCachingAtLevel(TestMultipleLevelCaching.java:113)
> at 
> org.apache.hadoop.mapred.TestMultipleLevelCaching.testMultiLevelCaching(TestMultipleLevelCaching.java:69)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4904) TestMultipleLevelCaching failed in barnch-1

[meng gong (JIRA)]

meng gong created MAPREDUCE-4904:


 Summary: TestMultipleLevelCaching failed in barnch-1
 Key: MAPREDUCE-4904
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4904
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: 1.2.0
Reporter: meng gong
Assignee: meng gong
 Fix For: 1.2.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4870) TestMRJobsWithHistoryService causes infinite loop if it fails

[Arun C Murthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun C Murthy updated MAPREDUCE-4870:
-

  Resolution: Fixed
   Fix Version/s: trunk-win
Target Version/s: trunk-win  (was: 3.0.0, trunk-win)
  Status: Resolved  (was: Patch Available)

I just committed this. Thanks Chris!

> TestMRJobsWithHistoryService causes infinite loop if it fails
> -
>
> Key: MAPREDUCE-4870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4870
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 3.0.0, trunk-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: MAPREDUCE-4870.1.patch
>
>
> {{TestMRJobsWithHistoryService#testJobHistoryData}} has a periodic poll and 
> sleep after job execution, checking for the application state to reach 
> {{RMAppState#FINISHED}}.  If the job fails, then the application could be in 
> a different terminal state, and this polling loop will never terminate.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-4869) TestMapReduceChildJVM fails in branch-trunk-win

[Arun C Murthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun C Murthy resolved MAPREDUCE-4869.
--

   Resolution: Fixed
Fix Version/s: trunk-win

I just committed this. Thanks Chris!

> TestMapReduceChildJVM fails in branch-trunk-win
> ---
>
> Key: MAPREDUCE-4869
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4869
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: trunk-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: MAPREDUCE-4869-branch-trunk-win.1.patch
>
>
> The YARN-233 patch for getting YARN working on Windows forgot to include a 
> corresponding change in {{TestMapReduceChildJVM}}, so the test is failing now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4397) Introduce HADOOP_SECURITY_CONF_DIR for task-controller

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540228#comment-13540228
 ] 

Matt Foley commented on MAPREDUCE-4397:
---

Patch accepted to 1.1.2.

> Introduce HADOOP_SECURITY_CONF_DIR for task-controller
> --
>
> Key: MAPREDUCE-4397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4397
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: task-controller
>Reporter: Luke Lu
>Assignee: Yu Gao
> Fix For: 1.1.2
>
> Attachments: mapreduce-4397-branch-1.patch, test-patch.result
>
>
> The linux task controller concurrently hard code the directory to look for 
> its config file at compile time (via the HADOOP_CONF_DIR macro). Adding a new 
> environment variable to look for task-controller's conf dir (with strict 
> permission checks) would make installation much more flexible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Closed] (MAPREDUCE-4397) Introduce HADOOP_SECURITY_CONF_DIR for task-controller

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley closed MAPREDUCE-4397.
-


> Introduce HADOOP_SECURITY_CONF_DIR for task-controller
> --
>
> Key: MAPREDUCE-4397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4397
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: task-controller
>Reporter: Luke Lu
>Assignee: Yu Gao
> Fix For: 1.1.2
>
> Attachments: mapreduce-4397-branch-1.patch, test-patch.result
>
>
> The linux task controller concurrently hard code the directory to look for 
> its config file at compile time (via the HADOOP_CONF_DIR macro). Adding a new 
> environment variable to look for task-controller's conf dir (with strict 
> permission checks) would make installation much more flexible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4397) Introduce HADOOP_SECURITY_CONF_DIR for task-controller

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated MAPREDUCE-4397:
--

Target Version/s: 1.1.2  (was: 1.2.0)

> Introduce HADOOP_SECURITY_CONF_DIR for task-controller
> --
>
> Key: MAPREDUCE-4397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4397
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: task-controller
>Reporter: Luke Lu
>Assignee: Yu Gao
> Fix For: 1.1.2
>
> Attachments: mapreduce-4397-branch-1.patch, test-patch.result
>
>
> The linux task controller concurrently hard code the directory to look for 
> its config file at compile time (via the HADOOP_CONF_DIR macro). Adding a new 
> environment variable to look for task-controller's conf dir (with strict 
> permission checks) would make installation much more flexible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4397) Introduce HADOOP_SECURITY_CONF_DIR for task-controller

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated MAPREDUCE-4397:
--

Fix Version/s: (was: 1.2.0)

> Introduce HADOOP_SECURITY_CONF_DIR for task-controller
> --
>
> Key: MAPREDUCE-4397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4397
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: task-controller
>Reporter: Luke Lu
>Assignee: Yu Gao
> Fix For: 1.1.2
>
> Attachments: mapreduce-4397-branch-1.patch, test-patch.result
>
>
> The linux task controller concurrently hard code the directory to look for 
> its config file at compile time (via the HADOOP_CONF_DIR macro). Adding a new 
> environment variable to look for task-controller's conf dir (with strict 
> permission checks) would make installation much more flexible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (MAPREDUCE-4272) SortedRanges.Range#compareTo is not spec compliant

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley reopened MAPREDUCE-4272:
---


> SortedRanges.Range#compareTo is not spec compliant
> --
>
> Key: MAPREDUCE-4272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4272
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: task
>Affects Versions: 1.0.3
>Reporter: Luke Lu
>Assignee: Yu Gao
>  Labels: alt-jdk
> Fix For: 1.1.2
>
> Attachments: mapreduce-4272-branch-1.0.patch, 
> mapreduce-4272-branch-2.patch, mapreduce-4272.patch
>
>
> SortedRanges.Range#compareTo does not satisfy the requirement of 
> Comparable#compareTo, where "the implementor must ensure 
> {noformat}sgn(x.compareTo(y)) == -sgn(y.compareTo(x)){noformat} for all x and 
> y."
> This is manifested as TestStreamingBadRecords failures in alternative JDKs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4272) SortedRanges.Range#compareTo is not spec compliant

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated MAPREDUCE-4272:
--

Target Version/s: 2.0.0-alpha, 1.1.2  (was: 1.2.0, 2.0.0-alpha)

> SortedRanges.Range#compareTo is not spec compliant
> --
>
> Key: MAPREDUCE-4272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4272
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: task
>Affects Versions: 1.0.3
>Reporter: Luke Lu
>Assignee: Yu Gao
>  Labels: alt-jdk
> Fix For: 1.1.2
>
> Attachments: mapreduce-4272-branch-1.0.patch, 
> mapreduce-4272-branch-2.patch, mapreduce-4272.patch
>
>
> SortedRanges.Range#compareTo does not satisfy the requirement of 
> Comparable#compareTo, where "the implementor must ensure 
> {noformat}sgn(x.compareTo(y)) == -sgn(y.compareTo(x)){noformat} for all x and 
> y."
> This is manifested as TestStreamingBadRecords failures in alternative JDKs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4272) SortedRanges.Range#compareTo is not spec compliant

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540227#comment-13540227
 ] 

Matt Foley commented on MAPREDUCE-4272:
---

Looks like the integration with trunk / 2.0 was not successful.  Keeping bug 
open for that.
Fix accepted into 1.1.2.

> SortedRanges.Range#compareTo is not spec compliant
> --
>
> Key: MAPREDUCE-4272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4272
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: task
>Affects Versions: 1.0.3
>Reporter: Luke Lu
>Assignee: Yu Gao
>  Labels: alt-jdk
> Fix For: 1.2.0, 1.1.2
>
> Attachments: mapreduce-4272-branch-1.0.patch, 
> mapreduce-4272-branch-2.patch, mapreduce-4272.patch
>
>
> SortedRanges.Range#compareTo does not satisfy the requirement of 
> Comparable#compareTo, where "the implementor must ensure 
> {noformat}sgn(x.compareTo(y)) == -sgn(y.compareTo(x)){noformat} for all x and 
> y."
> This is manifested as TestStreamingBadRecords failures in alternative JDKs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4272) SortedRanges.Range#compareTo is not spec compliant

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated MAPREDUCE-4272:
--

Fix Version/s: (was: 1.2.0)

> SortedRanges.Range#compareTo is not spec compliant
> --
>
> Key: MAPREDUCE-4272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4272
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: task
>Affects Versions: 1.0.3
>Reporter: Luke Lu
>Assignee: Yu Gao
>  Labels: alt-jdk
> Fix For: 1.1.2
>
> Attachments: mapreduce-4272-branch-1.0.patch, 
> mapreduce-4272-branch-2.patch, mapreduce-4272.patch
>
>
> SortedRanges.Range#compareTo does not satisfy the requirement of 
> Comparable#compareTo, where "the implementor must ensure 
> {noformat}sgn(x.compareTo(y)) == -sgn(y.compareTo(x)){noformat} for all x and 
> y."
> This is manifested as TestStreamingBadRecords failures in alternative JDKs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4813) AM timing out during job commit

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540226#comment-13540226
 ] 

Hadoop QA commented on MAPREDUCE-4813:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12562531/MAPREDUCE-4813-2-branch-0.23.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3179//console

This message is automatically generated.

> AM timing out during job commit
> ---
>
> Key: MAPREDUCE-4813
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: JobImplStateMachine.pdf, 
> MAPREDUCE-4813-2-branch-0.23.patch, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch
>
>
> The AM calls the output committer's {{commitJob}} method synchronously during 
> JobImpl state transitions, which means the JobImpl write lock is held the 
> entire time the job is being committed.  Holding the write lock prevents the 
> RM allocator thread from heartbeating to the RM.  Therefore if committing the 
> job takes too long (e.g.: the job has tons of files to commit and/or the 
> namenode is bogged down) then the AM appears to be unresponsive to the RM and 
> the RM kills the AM attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4813) AM timing out during job commit

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-4813:
--

Attachment: MAPREDUCE-4813-2-branch-0.23.patch

Thanks, Bobby.  Adding the patch for branch-0.23.

> AM timing out during job commit
> ---
>
> Key: MAPREDUCE-4813
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: JobImplStateMachine.pdf, 
> MAPREDUCE-4813-2-branch-0.23.patch, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch
>
>
> The AM calls the output committer's {{commitJob}} method synchronously during 
> JobImpl state transitions, which means the JobImpl write lock is held the 
> entire time the job is being committed.  Holding the write lock prevents the 
> RM allocator thread from heartbeating to the RM.  Therefore if committing the 
> job takes too long (e.g.: the job has tons of files to commit and/or the 
> namenode is bogged down) then the AM appears to be unresponsive to the RM and 
> the RM kills the AM attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4813) AM timing out during job commit

[Robert Joseph Evans (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540108#comment-13540108
 ] 

Robert Joseph Evans commented on MAPREDUCE-4813:


The new set of changes look good to me I am +1.

> AM timing out during job commit
> ---
>
> Key: MAPREDUCE-4813
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch
>
>
> The AM calls the output committer's {{commitJob}} method synchronously during 
> JobImpl state transitions, which means the JobImpl write lock is held the 
> entire time the job is being committed.  Holding the write lock prevents the 
> RM allocator thread from heartbeating to the RM.  Therefore if committing the 
> job takes too long (e.g.: the job has tons of files to commit and/or the 
> namenode is bogged down) then the AM appears to be unresponsive to the RM and 
> the RM kills the AM attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4894) Renewal / cancellation of JobHistory tokens

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated MAPREDUCE-4894:
--

Attachment: MR-4894_branch0.23.txt

branch-0.23 patch with unit test.

> Renewal / cancellation of JobHistory tokens
> ---
>
> Key: MAPREDUCE-4894
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4894
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.4
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
>Priority: Blocker
> Attachments: MAPREDUCE-4894_wip.txt, MR-4894_branch0.23.txt, 
> MR-4894_branch0.23.txt, MR-4894_trunk.txt, MR-4894.txt
>
>
> Equivalent of YARN-50 for JobHistory tokens.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4894) Renewal / cancellation of JobHistory tokens

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated MAPREDUCE-4894:
--

Attachment: MR-4894_trunk.txt

trunk patch. Requires YARN-50

> Renewal / cancellation of JobHistory tokens
> ---
>
> Key: MAPREDUCE-4894
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4894
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.4
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
>Priority: Blocker
> Attachments: MAPREDUCE-4894_wip.txt, MR-4894_branch0.23.txt, 
> MR-4894_branch0.23.txt, MR-4894_trunk.txt, MR-4894.txt
>
>
> Equivalent of YARN-50 for JobHistory tokens.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4808) Allow reduce-side merge to be pluggable

[Tendu Yogurtcu (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540092#comment-13540092
 ] 

Tendu Yogurtcu commented on MAPREDUCE-4808:
---

Thank you for your message. I am out of the office with limited access to 
e-mail, returning on Monday Jan 7th. Should you need immediate assistance, 
please contact Fernanda Tavares at 
ftava...@syncsort.com. Wishing you a Happy 
Holiday Season and a Happy New Year!



Regards,


> Allow reduce-side merge to be pluggable
> ---
>
> Key: MAPREDUCE-4808
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Affects Versions: 2.0.2-alpha
>Reporter: Arun C Murthy
>Assignee: Mariappan Asokan
> Fix For: 2.0.3-alpha
>
> Attachments: COMBO-mapreduce-4809-4812-4808.patch, 
> mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, 
> mapreduce-4808.patch, mapreduce-4808.patch, MergeManagerPlugin.pdf
>
>
> Allow reduce-side merge to be pluggable for MAPREDUCE-2454

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4808) Allow reduce-side merge to be pluggable

[Alejandro Abdelnur (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540091#comment-13540091
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4808:
---

+1, to the patch and design doc. 

I'll add that having this pluggability in place it would have allowed to have a 
fix for MAPREDUCE-4842 avail for jobs hitting the problem until the fix is 
delivered in a Hadoop release. 

I'll wait until after the New Year to give the opportunity to others to 
review/comment the patch and design doc.


> Allow reduce-side merge to be pluggable
> ---
>
> Key: MAPREDUCE-4808
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Affects Versions: 2.0.2-alpha
>Reporter: Arun C Murthy
>Assignee: Mariappan Asokan
> Fix For: 2.0.3-alpha
>
> Attachments: COMBO-mapreduce-4809-4812-4808.patch, 
> mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, 
> mapreduce-4808.patch, mapreduce-4808.patch, MergeManagerPlugin.pdf
>
>
> Allow reduce-side merge to be pluggable for MAPREDUCE-2454

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4813) AM timing out during job commit

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540088#comment-13540088
 ] 

Hadoop QA commented on MAPREDUCE-4813:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12562484/MAPREDUCE-4813-2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3178//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3178//console

This message is automatically generated.

> AM timing out during job commit
> ---
>
> Key: MAPREDUCE-4813
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch
>
>
> The AM calls the output committer's {{commitJob}} method synchronously during 
> JobImpl state transitions, which means the JobImpl write lock is held the 
> entire time the job is being committed.  Holding the write lock prevents the 
> RM allocator thread from heartbeating to the RM.  Therefore if committing the 
> job takes too long (e.g.: the job has tons of files to commit and/or the 
> namenode is bogged down) then the AM appears to be unresponsive to the RM and 
> the RM kills the AM attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4813) AM timing out during job commit

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540066#comment-13540066
 ] 

Hadoop QA commented on MAPREDUCE-4813:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12562484/MAPREDUCE-4813-2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3177//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3177//console

This message is automatically generated.

> AM timing out during job commit
> ---
>
> Key: MAPREDUCE-4813
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch
>
>
> The AM calls the output committer's {{commitJob}} method synchronously during 
> JobImpl state transitions, which means the JobImpl write lock is held the 
> entire time the job is being committed.  Holding the write lock prevents the 
> RM allocator thread from heartbeating to the RM.  Therefore if committing the 
> job takes too long (e.g.: the job has tons of files to commit and/or the 
> namenode is bogged down) then the AM appears to be unresponsive to the RM and 
> the RM kills the AM attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4813) AM timing out during job commit

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-4813:
--

Attachment: MAPREDUCE-4813-2.patch

Thanks for the review, Bobby.  Updated the patch to address the review comments:

* I went ahead and fixed the JOB_START in KILLED race since it's a one-line fix 
to ignore that event.
* I noticed that JOB_TASK_COMPLETED also needs to be handled in the FAIL_ABORT 
state.
* The commit cancel wait loop was busted, as it needs to recompute {{now}} 
within the loop to function properly.
* Standardized on StringUtils.stringifyException to report committer exceptions 
in diagnostics.  I had previously preserved the existing behavior, but I agree 
we should be consistent.
* Updated IOException to Exception within the try block of committer calls to 
make sure we don't "leak" exceptions and deadlock.

> AM timing out during job commit
> ---
>
> Key: MAPREDUCE-4813
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813.patch, MAPREDUCE-4813.patch, MAPREDUCE-4813.patch
>
>
> The AM calls the output committer's {{commitJob}} method synchronously during 
> JobImpl state transitions, which means the JobImpl write lock is held the 
> entire time the job is being committed.  Holding the write lock prevents the 
> RM allocator thread from heartbeating to the RM.  Therefore if committing the 
> job takes too long (e.g.: the job has tons of files to commit and/or the 
> namenode is bogged down) then the AM appears to be unresponsive to the RM and 
> the RM kills the AM attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3469) Port to 0.22 - Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes

[Karthik Kambatla (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540008#comment-13540008
 ] 

Karthik Kambatla commented on MAPREDUCE-3469:
-

This JIRA seems to have died down. Will close it in the next 24 hours, if no 
activity.

> Port to 0.22 - Implement limits on per-job JobConf, Counters, StatusReport, 
> Split-Sizes
> ---
>
> Key: MAPREDUCE-3469
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3469
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Mahadev konar
>Assignee: Konstantin Shvachko
>  Labels: critical-0.22.0
>
> We have come across issues in production clusters wherein users abuse 
> counters, statusreport messages and split sizes. One such case was when one 
> of the users had 100 million counters. This leads to jobtracker going out of 
> memory and being unresponsive. In this jira I am proposing to put sane limits 
> on the status report length, the number of counters and the size of block 
> locations returned by the input split. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4903) Job history server giving incorrect job start time.

[Nishan Shetty (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nishan Shetty updated MAPREDUCE-4903:
-

Attachment: MAPREDUCE-4903.PNG

> Job history server giving incorrect job start time.
> ---
>
> Key: MAPREDUCE-4903
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4903
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver
>Affects Versions: 2.0.1-alpha, 2.0.2-alpha
>Reporter: Nishan Shetty
> Attachments: MAPREDUCE-4903.PNG
>
>
> {code:xml}Job Name:  word count  
> User Name:  root  
> Queue:  default  
> State:  KILLED  
> Uberized:  false  
> Started:  Thu Jan 01 05:29:59 IST 1970  
> Finished:  Thu Dec 27 20:36:31 IST 2012  
> Elapsed:  376839hrs, 6mins, 31sec  
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4903) Job history server giving incorrect job start time.

[Nishan Shetty (JIRA)]

Nishan Shetty created MAPREDUCE-4903:


 Summary: Job history server giving incorrect job start time.
 Key: MAPREDUCE-4903
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4903
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 2.0.2-alpha, 2.0.1-alpha
Reporter: Nishan Shetty


{code:xml}Job Name:  word count  
User Name:  root  
Queue:  default  
State:  KILLED  
Uberized:  false  
Started:  Thu Jan 01 05:29:59 IST 1970  
Finished:  Thu Dec 27 20:36:31 IST 2012  
Elapsed:  376839hrs, 6mins, 31sec  
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4813) AM timing out during job commit

[Robert Joseph Evans (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539976#comment-13539976
 ] 

Robert Joseph Evans commented on MAPREDUCE-4813:


I have a few minor comments about CommitterEventHandler and one more serious 
one as well. When checking for the timeout we call context.getClock().getTime() 
twice.
{code}
// wait up to configured timeout for commit thread to finish
long timeoutTimestamp = context.getClock().getTime()
 + commitThreadCancelTimeoutMs;
long now = context.getClock().getTime();
{code}

I personally think it would be cleaner to call it once
{code}
// wait up to configured timeout for commit thread to finish
long now = context.getClock().getTime();
long timeoutTimestamp = now + commitThreadCancelTimeoutMs;
{code}

Also I noticed some inconsistencies in the error handling of various functions. 
In some places to get a message for an event we call 
{code}StringUtils.stringifyException(e){code} but in others it is just call 
{code}e.getMessage(){code} I am not sure if there is a reason for this or not, 
but I would prefer it to be consistent.

On a bit more serious note it looks like if we get any RuntimeException or 
Error while processing jobSetup, etc. the exception will be eaten.  We are not 
catching it and threadpool will likely just eat it.  This could result in a 
deadlock.  I would suggest at a minimum we catch all Exceptions instead of just 
IOException.


> AM timing out during job commit
> ---
>
> Key: MAPREDUCE-4813
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813.patch, 
> MAPREDUCE-4813.patch, MAPREDUCE-4813.patch
>
>
> The AM calls the output committer's {{commitJob}} method synchronously during 
> JobImpl state transitions, which means the JobImpl write lock is held the 
> entire time the job is being committed.  Holding the write lock prevents the 
> RM allocator thread from heartbeating to the RM.  Therefore if committing the 
> job takes too long (e.g.: the job has tons of files to commit and/or the 
> namenode is bogged down) then the AM appears to be unresponsive to the RM and 
> the RM kills the AM attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4813) AM timing out during job commit

[Robert Joseph Evans (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539974#comment-13539974
 ] 

Robert Joseph Evans commented on MAPREDUCE-4813:


I have gone through the state machine for JobImpl, and I think I have found a 
few issues.  I am not done with my review, but I wanted to give you a heads up 
on them.

It looks like it may be possible to get a JOB_START event after a JOB_KILL 
event.  This would be a very rare race condition where someone is trying to 
kill their job right after starting it.  I am not sure if it is even possible 
in practice, especially after this patch where the setup is removed from the 
critical path. So I would suggest that we file a minor JIRA to fix this at some 
point.

Also it looks like we need to handle JOB_MAP_TASK_RESCHEDULED and 
JOB_TASK_ATTEMPT_COMPLETED in the FAIL_ABORT state.  In the KILL_ABORT state we 
don't need to worry about them because the KILL_WAIT state should make sure all 
tasks have completed before going on, but there is no corresponding FAIL_WAIT 
state. 

> AM timing out during job commit
> ---
>
> Key: MAPREDUCE-4813
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4813
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: JobImplStateMachine.pdf, MAPREDUCE-4813-2.patch, 
> MAPREDUCE-4813-2.patch, MAPREDUCE-4813-2.patch, MAPREDUCE-4813.patch, 
> MAPREDUCE-4813.patch, MAPREDUCE-4813.patch
>
>
> The AM calls the output committer's {{commitJob}} method synchronously during 
> JobImpl state transitions, which means the JobImpl write lock is held the 
> entire time the job is being committed.  Holding the write lock prevents the 
> RM allocator thread from heartbeating to the RM.  Therefore if committing the 
> job takes too long (e.g.: the job has tons of files to commit and/or the 
> namenode is bogged down) then the AM appears to be unresponsive to the RM and 
> the RM kills the AM attempt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4902) Fix typo "receievd" should be "received" in log output

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539961#comment-13539961
 ] 

Hudson commented on MAPREDUCE-4902:
---

Integrated in Hadoop-Mapreduce-trunk #1297 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1297/])
MAPREDUCE-4902. Fix typo "receievd" should be "received" in log output. 
Contributed by Albert Chu (Revision 1426018)

 Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1426018
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java


> Fix typo "receievd" should be "received" in log output
> --
>
> Key: MAPREDUCE-4902
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Albert Chu
>Assignee: Albert Chu
>Priority: Trivial
> Fix For: 2.0.3-alpha, 0.23.6
>
> Attachments: MAPREDUCE-4902.patch
>
>
> Noticed a typo in the log output, "receievd" should be "received" 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher: for 
> url=8080/mapOutput?job=job_1356131733318_0002&reduce=0&map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0
>  sent hash and receievd reply

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4902) Fix typo "receievd" should be "received" in log output

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539950#comment-13539950
 ] 

Hudson commented on MAPREDUCE-4902:
---

Integrated in Hadoop-Hdfs-trunk #1267 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1267/])
MAPREDUCE-4902. Fix typo "receievd" should be "received" in log output. 
Contributed by Albert Chu (Revision 1426018)

 Result = FAILURE
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1426018
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java


> Fix typo "receievd" should be "received" in log output
> --
>
> Key: MAPREDUCE-4902
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Albert Chu
>Assignee: Albert Chu
>Priority: Trivial
> Fix For: 2.0.3-alpha, 0.23.6
>
> Attachments: MAPREDUCE-4902.patch
>
>
> Noticed a typo in the log output, "receievd" should be "received" 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher: for 
> url=8080/mapOutput?job=job_1356131733318_0002&reduce=0&map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0
>  sent hash and receievd reply

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4902) Fix typo "receievd" should be "received" in log output

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539936#comment-13539936
 ] 

Hudson commented on MAPREDUCE-4902:
---

Integrated in Hadoop-Hdfs-0.23-Build #476 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/476/])
svn merge -c 1426018 FIXES: MAPREDUCE-4902. Fix typo "receievd" should be 
"received" in log output. Contributed by Albert Chu (Revision 1426021)

 Result = UNSTABLE
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1426021
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java


> Fix typo "receievd" should be "received" in log output
> --
>
> Key: MAPREDUCE-4902
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Albert Chu
>Assignee: Albert Chu
>Priority: Trivial
> Fix For: 2.0.3-alpha, 0.23.6
>
> Attachments: MAPREDUCE-4902.patch
>
>
> Noticed a typo in the log output, "receievd" should be "received" 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher: for 
> url=8080/mapOutput?job=job_1356131733318_0002&reduce=0&map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0
>  sent hash and receievd reply

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4900) Dynamic configuration for task slots on TT

[Binglin Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Binglin Chang updated MAPREDUCE-4900:
-

Attachment: MAPREDUCE-4900-demo-with-JMX.v2.patch

fix bug, better handling the case that setting slot number is large than max 
slot number.

> Dynamic configuration for task slots on TT
> --
>
> Key: MAPREDUCE-4900
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4900
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>  Components: resourcemanager, tasktracker
>Affects Versions: 1.1.1
>Reporter: Junping Du
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-4900-demo.patch, 
> MAPREDUCE-4900-demo-with-JMX.patch, MAPREDUCE-4900-demo-with-JMX.v2.patch
>
>
> The current Hadoop MRV1 resource management logic assumes per node slot 
> number is static during the lifetime of the TT process. Allowing run-time 
> configuration on per node slot will give us finer granularity of resource 
> elasticity. This allows Hadoop workloads to coexist with other workloads on 
> the same hardware efficiently, whether or not the environment is virtualized.
> For more background or design details of this effort, please refer proposal 
> in HADOOP-9165.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4900) Dynamic configuration for task slots on TT

[Binglin Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Binglin Chang updated MAPREDUCE-4900:
-

Attachment: MAPREDUCE-4900-demo-with-JMX.patch

Add the same interface to JobTrackerMXBean to demonstrate JMX solution. 

> Dynamic configuration for task slots on TT
> --
>
> Key: MAPREDUCE-4900
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4900
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>  Components: resourcemanager, tasktracker
>Affects Versions: 1.1.1
>Reporter: Junping Du
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-4900-demo.patch, 
> MAPREDUCE-4900-demo-with-JMX.patch
>
>
> The current Hadoop MRV1 resource management logic assumes per node slot 
> number is static during the lifetime of the TT process. Allowing run-time 
> configuration on per node slot will give us finer granularity of resource 
> elasticity. This allows Hadoop workloads to coexist with other workloads on 
> the same hardware efficiently, whether or not the environment is virtualized.
> For more background or design details of this effort, please refer proposal 
> in HADOOP-9165.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira