[jira] [Updated] (MAPREDUCE-4720) Browser thinks History Server main page JS is taking too long
[ https://issues.apache.org/jira/browse/MAPREDUCE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-4720: Attachment: MAPREDUCE-4720.branch23.patch This is the final patch for branch-0.23. It adds deferred rendering to the HS jobs page and HS Job Tasks page. Both of the tables on these pages can become pretty big. @Luke: Could you please review and commit this patch? > Browser thinks History Server main page JS is taking too long > - > > Key: MAPREDUCE-4720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4720 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Robert Joseph Evans >Assignee: Ravi Prakash > Attachments: MAPREDUCE-4720.branch23.patch, MAPREDUCE-4720.patch, > MAPREDUCE-4720.patch > > > The main History Server page with the default settings of 20,000 jobs can > cause browsers to think that the JS on the page is stuck and ask you if you > want to kill it. This is a big usability problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4517) Too many INFO messages written out during AM to RM heartbeat
[
https://issues.apache.org/jira/browse/MAPREDUCE-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496020#comment-13496020
]
Hadoop QA commented on MAPREDUCE-4517:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553278/MAPREDUCE-4517.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3022//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3022//console
This message is automatically generated.
> Too many INFO messages written out during AM to RM heartbeat
>
>
> Key: MAPREDUCE-4517
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4517
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Affects Versions: 0.23.1, 2.0.1-alpha
>Reporter: James Kinley
>Assignee: Jason Lowe
>Priority: Minor
> Labels: patch
> Attachments: MAPREDUCE-4517.patch, MAPREDUCE-4517.patch,
> MAPREDUCE-4517.patch, MAPREDUCE-4517.patch
>
>
> Too many INFO log messages written out during AM to RM heartbeat. Based on
> default frequency of 1000ms (scheduler.heartbeat.interval-ms) either 2 or 4
> INFO messages are written out per second:
> LOG.info("Before Scheduling: " + getStat());
> List allocatedContainers = getResources();
> LOG.info("After Scheduling: " + getStat());
> if (allocatedContainers.size() > 0) {
> LOG.info("Before Assign: " + getStat());
> scheduledRequests.assign(allocatedContainers);
> LOG.info("After Assign: " + getStat());
> }
> These should probably be changed to DEBUG message to save the log growing too
> quickly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4517) Too many INFO messages written out during AM to RM heartbeat
[
https://issues.apache.org/jira/browse/MAPREDUCE-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4517:
--
Attachment: MAPREDUCE-4517.patch
Ack, I totally misunderstood what findbugs ws complaining about. My local
findbugs is happy with this version of the patch.
> Too many INFO messages written out during AM to RM heartbeat
>
>
> Key: MAPREDUCE-4517
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4517
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Affects Versions: 0.23.1, 2.0.1-alpha
>Reporter: James Kinley
>Assignee: Jason Lowe
>Priority: Minor
> Labels: patch
> Attachments: MAPREDUCE-4517.patch, MAPREDUCE-4517.patch,
> MAPREDUCE-4517.patch, MAPREDUCE-4517.patch
>
>
> Too many INFO log messages written out during AM to RM heartbeat. Based on
> default frequency of 1000ms (scheduler.heartbeat.interval-ms) either 2 or 4
> INFO messages are written out per second:
> LOG.info("Before Scheduling: " + getStat());
> List allocatedContainers = getResources();
> LOG.info("After Scheduling: " + getStat());
> if (allocatedContainers.size() > 0) {
> LOG.info("Before Assign: " + getStat());
> scheduledRequests.assign(allocatedContainers);
> LOG.info("After Assign: " + getStat());
> }
> These should probably be changed to DEBUG message to save the log growing too
> quickly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4517) Too many INFO messages written out during AM to RM heartbeat
[
https://issues.apache.org/jira/browse/MAPREDUCE-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495984#comment-13495984
]
Hadoop QA commented on MAPREDUCE-4517:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553271/MAPREDUCE-4517.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3021//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3021//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3021//console
This message is automatically generated.
> Too many INFO messages written out during AM to RM heartbeat
>
>
> Key: MAPREDUCE-4517
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4517
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Affects Versions: 0.23.1, 2.0.1-alpha
>Reporter: James Kinley
>Assignee: Jason Lowe
>Priority: Minor
> Labels: patch
> Attachments: MAPREDUCE-4517.patch, MAPREDUCE-4517.patch,
> MAPREDUCE-4517.patch
>
>
> Too many INFO log messages written out during AM to RM heartbeat. Based on
> default frequency of 1000ms (scheduler.heartbeat.interval-ms) either 2 or 4
> INFO messages are written out per second:
> LOG.info("Before Scheduling: " + getStat());
> List allocatedContainers = getResources();
> LOG.info("After Scheduling: " + getStat());
> if (allocatedContainers.size() > 0) {
> LOG.info("Before Assign: " + getStat());
> scheduledRequests.assign(allocatedContainers);
> LOG.info("After Assign: " + getStat());
> }
> These should probably be changed to DEBUG message to save the log growing too
> quickly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4517) Too many INFO messages written out during AM to RM heartbeat
[
https://issues.apache.org/jira/browse/MAPREDUCE-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4517:
--
Attachment: MAPREDUCE-4517.patch
Updated patch to fix findbug warnings. No unit test since it's just changing
logging behavior. I manually tested this with sleep jobs and verified it no
longer emits messages during heartbeats to the RM unless the scheduling state
has changed.
> Too many INFO messages written out during AM to RM heartbeat
>
>
> Key: MAPREDUCE-4517
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4517
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Affects Versions: 0.23.1, 2.0.1-alpha
>Reporter: James Kinley
>Assignee: Jason Lowe
>Priority: Minor
> Labels: patch
> Attachments: MAPREDUCE-4517.patch, MAPREDUCE-4517.patch,
> MAPREDUCE-4517.patch
>
>
> Too many INFO log messages written out during AM to RM heartbeat. Based on
> default frequency of 1000ms (scheduler.heartbeat.interval-ms) either 2 or 4
> INFO messages are written out per second:
> LOG.info("Before Scheduling: " + getStat());
> List allocatedContainers = getResources();
> LOG.info("After Scheduling: " + getStat());
> if (allocatedContainers.size() > 0) {
> LOG.info("Before Assign: " + getStat());
> scheduledRequests.assign(allocatedContainers);
> LOG.info("After Assign: " + getStat());
> }
> These should probably be changed to DEBUG message to save the log growing too
> quickly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4517) Too many INFO messages written out during AM to RM heartbeat
[
https://issues.apache.org/jira/browse/MAPREDUCE-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495953#comment-13495953
]
Hadoop QA commented on MAPREDUCE-4517:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553265/MAPREDUCE-4517.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3020//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3020//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3020//console
This message is automatically generated.
> Too many INFO messages written out during AM to RM heartbeat
>
>
> Key: MAPREDUCE-4517
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4517
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Affects Versions: 0.23.1, 2.0.1-alpha
>Reporter: James Kinley
>Assignee: Jason Lowe
>Priority: Minor
> Labels: patch
> Attachments: MAPREDUCE-4517.patch, MAPREDUCE-4517.patch
>
>
> Too many INFO log messages written out during AM to RM heartbeat. Based on
> default frequency of 1000ms (scheduler.heartbeat.interval-ms) either 2 or 4
> INFO messages are written out per second:
> LOG.info("Before Scheduling: " + getStat());
> List allocatedContainers = getResources();
> LOG.info("After Scheduling: " + getStat());
> if (allocatedContainers.size() > 0) {
> LOG.info("Before Assign: " + getStat());
> scheduledRequests.assign(allocatedContainers);
> LOG.info("After Assign: " + getStat());
> }
> These should probably be changed to DEBUG message to save the log growing too
> quickly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4517) Too many INFO messages written out during AM to RM heartbeat
[
https://issues.apache.org/jira/browse/MAPREDUCE-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4517:
--
Attachment: MAPREDUCE-4517.patch
Patch to only log scheduling stats when they change.
> Too many INFO messages written out during AM to RM heartbeat
>
>
> Key: MAPREDUCE-4517
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4517
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Affects Versions: 0.23.1, 2.0.1-alpha
>Reporter: James Kinley
>Priority: Minor
> Labels: patch
> Attachments: MAPREDUCE-4517.patch, MAPREDUCE-4517.patch
>
>
> Too many INFO log messages written out during AM to RM heartbeat. Based on
> default frequency of 1000ms (scheduler.heartbeat.interval-ms) either 2 or 4
> INFO messages are written out per second:
> LOG.info("Before Scheduling: " + getStat());
> List allocatedContainers = getResources();
> LOG.info("After Scheduling: " + getStat());
> if (allocatedContainers.size() > 0) {
> LOG.info("Before Assign: " + getStat());
> scheduledRequests.assign(allocatedContainers);
> LOG.info("After Assign: " + getStat());
> }
> These should probably be changed to DEBUG message to save the log growing too
> quickly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4517) Too many INFO messages written out during AM to RM heartbeat
[
https://issues.apache.org/jira/browse/MAPREDUCE-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4517:
--
Fix Version/s: (was: trunk)
Assignee: Jason Lowe
Target Version/s: 2.0.3-alpha, 0.23.5
Affects Version/s: 0.23.1
2.0.1-alpha
Status: Patch Available (was: Open)
> Too many INFO messages written out during AM to RM heartbeat
>
>
> Key: MAPREDUCE-4517
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4517
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Affects Versions: 2.0.1-alpha, 0.23.1
>Reporter: James Kinley
>Assignee: Jason Lowe
>Priority: Minor
> Labels: patch
> Attachments: MAPREDUCE-4517.patch, MAPREDUCE-4517.patch
>
>
> Too many INFO log messages written out during AM to RM heartbeat. Based on
> default frequency of 1000ms (scheduler.heartbeat.interval-ms) either 2 or 4
> INFO messages are written out per second:
> LOG.info("Before Scheduling: " + getStat());
> List allocatedContainers = getResources();
> LOG.info("After Scheduling: " + getStat());
> if (allocatedContainers.size() > 0) {
> LOG.info("Before Assign: " + getStat());
> scheduledRequests.assign(allocatedContainers);
> LOG.info("After Assign: " + getStat());
> }
> These should probably be changed to DEBUG message to save the log growing too
> quickly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4791) Javadoc for KeyValueTextInputFormat should include default separator and how to change it
Matt Lavin created MAPREDUCE-4791: - Summary: Javadoc for KeyValueTextInputFormat should include default separator and how to change it Key: MAPREDUCE-4791 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4791 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 1.1.0 Reporter: Matt Lavin Priority: Minor The javadoc for KeyValueTextInputFormat says "Each line is divided into key and value parts by a separator byte" but it doesn't say what the separator byte is or how to change it. After some exploration I noticed that the default is the Tab character and that the value can be changed by the JobConf value "key.value.separator.in.input.line" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495804#comment-13495804 ] Alejandro Abdelnur commented on MAPREDUCE-2454: --- +1 > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha >Reporter: Mariappan Asokan >Assignee: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, > KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mr-2454-on-mr-279-build82.patch.gz, > MR-2454-trunkPatchPreview.gz, ReduceInputSorter.java > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2454) Allow external sorter plugin for MR
[
https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495730#comment-13495730
]
Hadoop QA commented on MAPREDUCE-2454:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553200/mapreduce-2454.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3019//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3019//console
This message is automatically generated.
> Allow external sorter plugin for MR
> ---
>
> Key: MAPREDUCE-2454
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha
>Reporter: Mariappan Asokan
>Assignee: Mariappan Asokan
>Priority: Minor
> Labels: features, performance, plugin, sort
> Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf,
> KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mr-2454-on-mr-279-build82.patch.gz,
> MR-2454-trunkPatchPreview.gz, ReduceInputSorter.java
>
>
> Define interfaces and some abstract classes in the Hadoop framework to
> facilitate external sorter plugins both on the Map and Reduce sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariappan Asokan updated MAPREDUCE-2454: Status: Patch Available (was: Open) > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.2-alpha, 2.0.0-alpha, 3.0.0 >Reporter: Mariappan Asokan >Assignee: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, > KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mr-2454-on-mr-279-build82.patch.gz, > MR-2454-trunkPatchPreview.gz, ReduceInputSorter.java > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariappan Asokan updated MAPREDUCE-2454: Attachment: mapreduce-2454.patch Hi Alejandro, Thanks for the pointing out the nits. I fixed all of them. I am uploading a new patch. -- Asokan > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha >Reporter: Mariappan Asokan >Assignee: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, > KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mr-2454-on-mr-279-build82.patch.gz, > MR-2454-trunkPatchPreview.gz, ReduceInputSorter.java > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariappan Asokan updated MAPREDUCE-2454: Status: Open (was: Patch Available) > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.2-alpha, 2.0.0-alpha, 3.0.0 >Reporter: Mariappan Asokan >Assignee: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, > KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz, > ReduceInputSorter.java > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495660#comment-13495660 ] Alejandro Abdelnur edited comment on MAPREDUCE-2454 at 11/12/12 9:42 PM: - Asokan, Thanks for patience working out the design and implementation of this JIRA. The latest patch looks good and it addresses the concerns voiced regarding big changes that could destabilize the MR framework. There are few minor NITs that should be addressed in the patch (at the end of this comment). +1 after these NITs are addressed and jenkins test-patch OKs the new patch. Patch NITs: * PreReduceProcessor.java: unused import: Shuffle * ReduceTasks.java: line 357, PreReduceProcessor merger = (PreReduceProcessor) ReflectionUtils.newInstance(..., no need for the "(PreReduceProcessor)" casting. * Shuffle.java, unused imports: FileSystem, LocalDirAllocator, CompressionCodec, MapOutputFile, RawKeyValueIterator, Reducer, CombineOutputCollector * TestLimitNQuery.java, unused imports: FileInputStream, FileNotFoundException, FileStatus, JobCounter, TaskAttemptID, ReflectionUtils was (Author: tucu00): Asokan, Thanks for patience working out the design and implementation of this JIRA. The latest patch looks good and it addresses the concerns voiced regarding big changes that could destabilize the MR framework. There are few minor NITs that should be addressed in the patch (at the end of this comment). +1 after these NITs are addresses and jenkins test-patch OKs the new patch. Patch NITs: * PreReduceProcessor.java: unused import: Shuffle * ReduceTasks.java: line 357, PreReduceProcessor merger = (PreReduceProcessor) ReflectionUtils.newInstance(..., no need for the "(PreReduceProcessor)" casting. * Shuffle.java, unused imports: FileSystem, LocalDirAllocator, CompressionCodec, MapOutputFile, RawKeyValueIterator, Reducer, CombineOutputCollector * TestLimitNQuery.java, unused imports: FileInputStream, FileNotFoundException, FileStatus, JobCounter, TaskAttemptID, ReflectionUtils > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha >Reporter: Mariappan Asokan >Assignee: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, > KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz, > ReduceInputSorter.java > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495660#comment-13495660 ] Alejandro Abdelnur commented on MAPREDUCE-2454: --- Asokan, Thanks for patience working out the design and implementation of this JIRA. The latest patch looks good and it addresses the concerns voiced regarding big changes that could destabilize the MR framework. There are few minor NITs that should be addressed in the patch (at the end of this comment). +1 after these NITs are addresses and jenkins test-patch OKs the new patch. Patch NITs: * PreReduceProcessor.java: unused import: Shuffle * ReduceTasks.java: line 357, PreReduceProcessor merger = (PreReduceProcessor) ReflectionUtils.newInstance(..., no need for the "(PreReduceProcessor)" casting. * Shuffle.java, unused imports: FileSystem, LocalDirAllocator, CompressionCodec, MapOutputFile, RawKeyValueIterator, Reducer, CombineOutputCollector * TestLimitNQuery.java, unused imports: FileInputStream, FileNotFoundException, FileStatus, JobCounter, TaskAttemptID, ReflectionUtils > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha >Reporter: Mariappan Asokan >Assignee: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, > KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz, > ReduceInputSorter.java > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4790) MapReduce build script would be more readable using abspath
[ https://issues.apache.org/jira/browse/MAPREDUCE-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated MAPREDUCE-4790: - Attachment: MAPREDUCE-4790-branch-trunk-win.patch > MapReduce build script would be more readable using abspath > --- > > Key: MAPREDUCE-4790 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4790 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: MAPREDUCE-4790-branch-trunk-win.patch > > > This is a follow-up to MAPREDUCE-4780, which was resolved before addressing > some feedback to use abspath instead of normpath for improved readability in > the build script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4786) Job End Notification retry interval is 5 milliseconds by default
[
https://issues.apache.org/jira/browse/MAPREDUCE-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4786:
---
Resolution: Fixed
Fix Version/s: 0.23.5
2.0.3-alpha
3.0.0
Status: Resolved (was: Patch Available)
Thanks Ravi,
I pulled this into trunk, branch-2, and branch-0.23
> Job End Notification retry interval is 5 milliseconds by default
>
>
> Key: MAPREDUCE-4786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4786
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.2-alpha, 0.23.4
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Fix For: 3.0.0, 2.0.3-alpha, 0.23.5
>
> Attachments: MAPREDUCE-4786.branch-2.patch,
> MAPREDUCE-4786.branch-2.patch, MAPREDUCE-4786.patch, MAPREDUCE-4786.patch,
> MAPREDUCE-4786.patch
>
>
> Courtesy [~stevenwillis] and [~qwertymaniac]
> {quote}
> From: Harsh J
> I believe the configs of the latter of both of the above
> classifications were meant to be added in as replacement names, but
> the property names got added in wrong (as the former/older named ones)
> in the XML.
> the word "seconds" in the description of retries? The code in MR2's
> JobEndNotifier seems to expect seconds but uses it directly in
> Thread.sleep(…) without making it milliseconds, which may be a bug we
> need to fix as well, perhaps in a same issue as the configs ones.
> On Fri, Nov 9, 2012 at 11:21 PM, Steven Willis wrote:
> > And I noticed that there are some duplicate properties with different
> > values and different descriptions:
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4790) MapReduce build script would be more readable using abspath
Chris Nauroth created MAPREDUCE-4790: Summary: MapReduce build script would be more readable using abspath Key: MAPREDUCE-4790 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4790 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth This is a follow-up to MAPREDUCE-4780, which was resolved before addressing some feedback to use abspath instead of normpath for improved readability in the build script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4786) Job End Notification retry interval is 5 milliseconds by default
[
https://issues.apache.org/jira/browse/MAPREDUCE-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495613#comment-13495613
]
Hudson commented on MAPREDUCE-4786:
---
Integrated in Hadoop-trunk-Commit #3003 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3003/])
MAPREDUCE-4786. Job End Notification retry interval is 5 milliseconds by
default (Ravi Prakash via bobby) (Revision 1408444)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408444
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
> Job End Notification retry interval is 5 milliseconds by default
>
>
> Key: MAPREDUCE-4786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4786
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.2-alpha, 0.23.4
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4786.branch-2.patch,
> MAPREDUCE-4786.branch-2.patch, MAPREDUCE-4786.patch, MAPREDUCE-4786.patch,
> MAPREDUCE-4786.patch
>
>
> Courtesy [~stevenwillis] and [~qwertymaniac]
> {quote}
> From: Harsh J
> I believe the configs of the latter of both of the above
> classifications were meant to be added in as replacement names, but
> the property names got added in wrong (as the former/older named ones)
> in the XML.
> the word "seconds" in the description of retries? The code in MR2's
> JobEndNotifier seems to expect seconds but uses it directly in
> Thread.sleep(…) without making it milliseconds, which may be a bug we
> need to fix as well, perhaps in a same issue as the configs ones.
> On Fri, Nov 9, 2012 at 11:21 PM, Steven Willis wrote:
> > And I noticed that there are some duplicate properties with different
> > values and different descriptions:
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4786) Job End Notification retry interval is 5 milliseconds by default
[
https://issues.apache.org/jira/browse/MAPREDUCE-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495600#comment-13495600
]
Robert Joseph Evans commented on MAPREDUCE-4786:
Oh I forgot to mention that I ran the tests that failed manually and they
passed for me. I am not really sure what happened there. They look like there
are some races in those tests.
> Job End Notification retry interval is 5 milliseconds by default
>
>
> Key: MAPREDUCE-4786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4786
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.2-alpha, 0.23.4
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4786.branch-2.patch,
> MAPREDUCE-4786.branch-2.patch, MAPREDUCE-4786.patch, MAPREDUCE-4786.patch,
> MAPREDUCE-4786.patch
>
>
> Courtesy [~stevenwillis] and [~qwertymaniac]
> {quote}
> From: Harsh J
> I believe the configs of the latter of both of the above
> classifications were meant to be added in as replacement names, but
> the property names got added in wrong (as the former/older named ones)
> in the XML.
> the word "seconds" in the description of retries? The code in MR2's
> JobEndNotifier seems to expect seconds but uses it directly in
> Thread.sleep(…) without making it milliseconds, which may be a bug we
> need to fix as well, perhaps in a same issue as the configs ones.
> On Fri, Nov 9, 2012 at 11:21 PM, Steven Willis wrote:
> > And I noticed that there are some duplicate properties with different
> > values and different descriptions:
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4786) Job End Notification retry interval is 5 milliseconds by default
[
https://issues.apache.org/jira/browse/MAPREDUCE-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495594#comment-13495594
]
Robert Joseph Evans commented on MAPREDUCE-4786:
Better, but the value of mapreduce.job.end-notification.url needs to be removed
or commented out. I can do that when I check it in. +1
> Job End Notification retry interval is 5 milliseconds by default
>
>
> Key: MAPREDUCE-4786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4786
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.2-alpha, 0.23.4
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4786.branch-2.patch,
> MAPREDUCE-4786.branch-2.patch, MAPREDUCE-4786.patch, MAPREDUCE-4786.patch,
> MAPREDUCE-4786.patch
>
>
> Courtesy [~stevenwillis] and [~qwertymaniac]
> {quote}
> From: Harsh J
> I believe the configs of the latter of both of the above
> classifications were meant to be added in as replacement names, but
> the property names got added in wrong (as the former/older named ones)
> in the XML.
> the word "seconds" in the description of retries? The code in MR2's
> JobEndNotifier seems to expect seconds but uses it directly in
> Thread.sleep(…) without making it milliseconds, which may be a bug we
> need to fix as well, perhaps in a same issue as the configs ones.
> On Fri, Nov 9, 2012 at 11:21 PM, Steven Willis wrote:
> > And I noticed that there are some duplicate properties with different
> > values and different descriptions:
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2454) Allow external sorter plugin for MR
[
https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1349#comment-1349
]
Hadoop QA commented on MAPREDUCE-2454:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553151/mapreduce-2454.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3017//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3017//console
This message is automatically generated.
> Allow external sorter plugin for MR
> ---
>
> Key: MAPREDUCE-2454
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha
>Reporter: Mariappan Asokan
>Assignee: Mariappan Asokan
>Priority: Minor
> Labels: features, performance, plugin, sort
> Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf,
> KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz,
> ReduceInputSorter.java
>
>
> Define interfaces and some abstract classes in the Hadoop framework to
> facilitate external sorter plugins both on the Map and Reduce sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4780) MapReduce distribution build fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved MAPREDUCE-4780. Resolution: Fixed Fix Version/s: trunk-win Hadoop Flags: Reviewed +1. Committed the patch to branch-trunk-win. Thank you Chris. > MapReduce distribution build fails on Windows > - > > Key: MAPREDUCE-4780 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4780 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: trunk-win > > Attachments: MAPREDUCE-4780-branch-trunk-win.patch > > > Distribution build relies on sh scripts that do not work on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4786) Job End Notification retry interval is 5 milliseconds by default
[
https://issues.apache.org/jira/browse/MAPREDUCE-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495549#comment-13495549
]
Hadoop QA commented on MAPREDUCE-4786:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553161/MAPREDUCE-4786.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.mapreduce.v2.app.TestRecovery
org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3018//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3018//console
This message is automatically generated.
> Job End Notification retry interval is 5 milliseconds by default
>
>
> Key: MAPREDUCE-4786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4786
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.2-alpha, 0.23.4
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4786.branch-2.patch,
> MAPREDUCE-4786.branch-2.patch, MAPREDUCE-4786.patch, MAPREDUCE-4786.patch,
> MAPREDUCE-4786.patch
>
>
> Courtesy [~stevenwillis] and [~qwertymaniac]
> {quote}
> From: Harsh J
> I believe the configs of the latter of both of the above
> classifications were meant to be added in as replacement names, but
> the property names got added in wrong (as the former/older named ones)
> in the XML.
> the word "seconds" in the description of retries? The code in MR2's
> JobEndNotifier seems to expect seconds but uses it directly in
> Thread.sleep(…) without making it milliseconds, which may be a bug we
> need to fix as well, perhaps in a same issue as the configs ones.
> On Fri, Nov 9, 2012 at 11:21 PM, Steven Willis wrote:
> > And I noticed that there are some duplicate properties with different
> > values and different descriptions:
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4789) Map Reduce Counters are 0 on the Job Tracker job details page
[ https://issues.apache.org/jira/browse/MAPREDUCE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495545#comment-13495545 ] Karthik Kambatla commented on MAPREDUCE-4789: - Sorry, my bad. The zeroes are just the job counters. > Map Reduce Counters are 0 on the Job Tracker job details page > - > > Key: MAPREDUCE-4789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Jeff Lord > Attachments: wordcount-on-etc-hadoop.png > > > Counters on the jobtracker details page are 0 for map and reduce columns. > This is reproduced using a simple wordcount. Of note is that customer > counters show properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4789) Map Reduce Counters are 0 on the Job Tracker job details page
[ https://issues.apache.org/jira/browse/MAPREDUCE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla resolved MAPREDUCE-4789. - Resolution: Invalid Closing this as invalid, MR counters are non-zero as in the attachment. > Map Reduce Counters are 0 on the Job Tracker job details page > - > > Key: MAPREDUCE-4789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Jeff Lord > Attachments: wordcount-on-etc-hadoop.png > > > Counters on the jobtracker details page are 0 for map and reduce columns. > This is reproduced using a simple wordcount. Of note is that customer > counters show properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4789) Map Reduce Counters are 0 on the Job Tracker job details page
[ https://issues.apache.org/jira/browse/MAPREDUCE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reopened MAPREDUCE-4789: - Re-opening as the problem exists in branch-1. The version tagged here might be wrong, but the problem seems to exist. See the attachment. > Map Reduce Counters are 0 on the Job Tracker job details page > - > > Key: MAPREDUCE-4789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Jeff Lord > Attachments: wordcount-on-etc-hadoop.png > > > Counters on the jobtracker details page are 0 for map and reduce columns. > This is reproduced using a simple wordcount. Of note is that customer > counters show properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4751) AM stuck in KILL_WAIT for days
[ https://issues.apache.org/jira/browse/MAPREDUCE-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495537#comment-13495537 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4751: Thanks for all the help, Bobby and Jason! > AM stuck in KILL_WAIT for days > -- > > Key: MAPREDUCE-4751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3, 2.0.2-alpha >Reporter: Ravi Prakash >Assignee: Vinod Kumar Vavilapalli > Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4751-20121108.txt, > MAPREDUCE-4751-20121109.txt, MR-4751-branch-0.23.txt, > TaskAttemptStateGraph.jpg > > > We found some jobs were stuck in KILL_WAIT for days on end. The RM shows them > as RUNNING. When you go to the AM, it shows it in the KILL_WAIT state, and a > few maps running. All these maps were scheduled on nodes which are now in the > RM's Lost nodes list. The running maps are in the FAIL_CONTAINER_CLEANUP state -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4789) Map Reduce Counters are 0 on the Job Tracker job details page
[ https://issues.apache.org/jira/browse/MAPREDUCE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4789: Attachment: wordcount-on-etc-hadoop.png > Map Reduce Counters are 0 on the Job Tracker job details page > - > > Key: MAPREDUCE-4789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Jeff Lord > Attachments: wordcount-on-etc-hadoop.png > > > Counters on the jobtracker details page are 0 for map and reduce columns. > This is reproduced using a simple wordcount. Of note is that customer > counters show properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4789) Map Reduce Counters are 0 on the Job Tracker job details page
[ https://issues.apache.org/jira/browse/MAPREDUCE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4789: Affects Version/s: (was: 2.0.0-alpha) 1.1.0 > Map Reduce Counters are 0 on the Job Tracker job details page > - > > Key: MAPREDUCE-4789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Jeff Lord > Attachments: wordcount-on-etc-hadoop.png > > > Counters on the jobtracker details page are 0 for map and reduce columns. > This is reproduced using a simple wordcount. Of note is that customer > counters show properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4786) Job End Notification retry interval is 5 milliseconds by default
[
https://issues.apache.org/jira/browse/MAPREDUCE-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-4786:
Attachment: MAPREDUCE-4786.patch
Same as patch as before except with --no-prefix
> Job End Notification retry interval is 5 milliseconds by default
>
>
> Key: MAPREDUCE-4786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4786
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.2-alpha, 0.23.4
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4786.branch-2.patch,
> MAPREDUCE-4786.branch-2.patch, MAPREDUCE-4786.patch, MAPREDUCE-4786.patch,
> MAPREDUCE-4786.patch
>
>
> Courtesy [~stevenwillis] and [~qwertymaniac]
> {quote}
> From: Harsh J
> I believe the configs of the latter of both of the above
> classifications were meant to be added in as replacement names, but
> the property names got added in wrong (as the former/older named ones)
> in the XML.
> the word "seconds" in the description of retries? The code in MR2's
> JobEndNotifier seems to expect seconds but uses it directly in
> Thread.sleep(…) without making it milliseconds, which may be a bug we
> need to fix as well, perhaps in a same issue as the configs ones.
> On Fri, Nov 9, 2012 at 11:21 PM, Steven Willis wrote:
> > And I noticed that there are some duplicate properties with different
> > values and different descriptions:
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4786) Job End Notification retry interval is 5 milliseconds by default
[
https://issues.apache.org/jira/browse/MAPREDUCE-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-4786:
Attachment: MAPREDUCE-4786.patch
MAPREDUCE-4786.branch-2.patch
All right Bobby!
Here you go
> Job End Notification retry interval is 5 milliseconds by default
>
>
> Key: MAPREDUCE-4786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4786
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.2-alpha, 0.23.4
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4786.branch-2.patch,
> MAPREDUCE-4786.branch-2.patch, MAPREDUCE-4786.patch, MAPREDUCE-4786.patch,
> MAPREDUCE-4786.patch
>
>
> Courtesy [~stevenwillis] and [~qwertymaniac]
> {quote}
> From: Harsh J
> I believe the configs of the latter of both of the above
> classifications were meant to be added in as replacement names, but
> the property names got added in wrong (as the former/older named ones)
> in the XML.
> the word "seconds" in the description of retries? The code in MR2's
> JobEndNotifier seems to expect seconds but uses it directly in
> Thread.sleep(…) without making it milliseconds, which may be a bug we
> need to fix as well, perhaps in a same issue as the configs ones.
> On Fri, Nov 9, 2012 at 11:21 PM, Steven Willis wrote:
> > And I noticed that there are some duplicate properties with different
> > values and different descriptions:
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4780) MapReduce distribution build fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495521#comment-13495521 ] Raja Aluri commented on MAPREDUCE-4780: --- +1 LGTM. > MapReduce distribution build fails on Windows > - > > Key: MAPREDUCE-4780 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4780 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: MAPREDUCE-4780-branch-trunk-win.patch > > > Distribution build relies on sh scripts that do not work on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4425) Speculation + Fetch failures can lead to a hung job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495510#comment-13495510 ] Robert Joseph Evans commented on MAPREDUCE-4425: Thanks for the new patch. It looks good +1. > Speculation + Fetch failures can lead to a hung job > --- > > Key: MAPREDUCE-4425 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4425 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Siddharth Seth >Assignee: Jason Lowe >Priority: Critical > Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4425-branch23.patch, MAPREDUCE-4425.patch, > MAPREDUCE-4425.patch > > > After a task goes to SUCCEEDED, FAILED/KILLED attempts are ignored. > 1. attemp1 starts > 2. speculative attempt starts > 3. attempt 1 completes - Task moves to SUCCEEDED state > 4. speculative attempt is KILLED > 5. T_ATTEMPT_KILLED is ignored. > 6. attemp1 1 fails with TOO_MANY_FETCH_FAILURES > The job will effectively hang, since a new task attempt isn't started. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4425) Speculation + Fetch failures can lead to a hung job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4425: --- Resolution: Fixed Fix Version/s: 0.23.5 2.0.3-alpha 3.0.0 Status: Resolved (was: Patch Available) Thanks again for fixing this, I put it into trunk, branch-2, and branch-0.23 > Speculation + Fetch failures can lead to a hung job > --- > > Key: MAPREDUCE-4425 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4425 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Siddharth Seth >Assignee: Jason Lowe >Priority: Critical > Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4425-branch23.patch, MAPREDUCE-4425.patch, > MAPREDUCE-4425.patch > > > After a task goes to SUCCEEDED, FAILED/KILLED attempts are ignored. > 1. attemp1 starts > 2. speculative attempt starts > 3. attempt 1 completes - Task moves to SUCCEEDED state > 4. speculative attempt is KILLED > 5. T_ATTEMPT_KILLED is ignored. > 6. attemp1 1 fails with TOO_MANY_FETCH_FAILURES > The job will effectively hang, since a new task attempt isn't started. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4789) Map Reduce Counters are 0 on the Job Tracker job details page
[ https://issues.apache.org/jira/browse/MAPREDUCE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4789. -- Resolution: Invalid We don't support MRv1 in hadoop-2.x. > Map Reduce Counters are 0 on the Job Tracker job details page > - > > Key: MAPREDUCE-4789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 2.0.0-alpha >Reporter: Jeff Lord > > Counters on the jobtracker details page are 0 for map and reduce columns. > This is reproduced using a simple wordcount. Of note is that customer > counters show properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariappan Asokan updated MAPREDUCE-2454: Status: Patch Available (was: Open) > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.2-alpha, 2.0.0-alpha, 3.0.0 >Reporter: Mariappan Asokan >Assignee: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, > KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz, > ReduceInputSorter.java > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2454) Allow external sorter plugin for MR
[
https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mariappan Asokan updated MAPREDUCE-2454:
Attachment: mapreduce-2454.patch
Hi Alejandro,
I made minor changes in annotations. Also, I am summarizing the visibility
changes in the following table(Hope it is formatted correctly. If not I will
attach a separate file to this Jira):
||Class||Protection||Annotation||Reason||
|{{org.apache.hadoop.mapred.MapTask}}|Changed from package protected to
public|Annotated as Public, Unstable|To be able to access inner interface
{{PostMapProcessor}} and inner class {{MapOutputBuffer}} from external
plug-ins.|
|{{org.apache.hadoop.mapred.MapTask.MapOutputCollector}}|Name changed to
PostMapProcessor and protection changed from package protected to
public|Annotated as Public, Unstable|To be able to implement external plug-ins.|
|{{org.apache.hadoop.mapred.MapTask.MapOutputBuffer}}|Changed to a static inner
class and protection changed from package protected to public|Annotated as
Public, Unstable|To be able to use the default implementation of
{{PostMapProcessor}}.|
|{{org.apache.hadoop.mapred.SpillRecord}}|Changed from package protected to
public|Annotated as Public, Unstable|To be able to write the intermediate
output from {{PostMapProcessor}} plug-ins.|
|{{org.apache.hadoop.mapred.Task.CombinerRunner}}|Changed from protected to
public|Changed from Private, Unstable to Public, Unstable|For code reuse to run
the {{Combiner}} from external {{PostMapProcessor}} plug-ins.|
|{{org.apache.hadoop.mapred.Task.TaskReporter}}|Changed from protected to
public|Changed from Private, Unstable to Public, Unstable|Has to be passed to
{{CombinerRunner}} from external {{PostMapProcessor}} plug-ins.|
|{{org.apache.hadoop.mapreduce.task.reduce.ExceptionReporter}}|Changed from
package protected to public|Annotated as Public, Unstable|To report exception
to {{Shuffle}} from {{PreReduceProcessor.}}|
|{{org.apache.hadoop.mapreduce.task.reduce.MapHost}}|Changed from package
protected to public|Annotated as Public, Unstable|Passed to {{shuffle()}}
method in {{MapOutput.}}|
|{{org.apache.hadoop.mapreduce.task.reduce.MapOutput}}|Changed from package
protected to public|Annotated as Public, Unstable|Returned from {{reserve()}}
method in {{PreReduceProcessor.}}|
|{{org.apache.hadoop.mapreduce.task.reduce.MergeManager}}|No change|Changed
from Private, Unstable to Public, Unstable|To reuse the default implementation
of {{PreReduceProcessor.}}|
|{{org.apache.hadoop.mapreduce.task.reduce.Shuffle}}|No change|Changed from
Private, Unstable to Public, Unstable|Passed to external {{PreReduceProcessor}}
plug-ins.|
|{{org.apache.hadoop.mapreduce.task.reduce.ShuffleClientMetrics}}|Changed from
package protected to public|Annotated as Public, Unstable|Passed to
{{shuffle()}} method in external {{MapOutput.}}|
> Allow external sorter plugin for MR
> ---
>
> Key: MAPREDUCE-2454
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha
>Reporter: Mariappan Asokan
>Assignee: Mariappan Asokan
>Priority: Minor
> Labels: features, performance, plugin, sort
> Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf,
> KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz,
> ReduceInputSorter.java
>
>
> Define interfaces and some abstract classes in the Hadoop framework to
> facilitate external sorter plugins both on the Map and Reduce sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4781) Unit test TestKerberosAuthenticationHandler fails with ant 1.8.3+
[
https://issues.apache.org/jira/browse/MAPREDUCE-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495491#comment-13495491
]
Hadoop QA commented on MAPREDUCE-4781:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553149/HADOOP-9024-branch-1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3016//console
This message is automatically generated.
> Unit test TestKerberosAuthenticationHandler fails with ant 1.8.3+
> -
>
> Key: MAPREDUCE-4781
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4781
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 1.0.3
> Environment: Fedora 17_64 on x86
>Reporter: Amir Sanjar
> Attachments: HADOOP-9024-branch-1.patch, MAPREDUCE-4781-branch-1.patch
>
>
> Problem:
> JUnit tag @Ignore is not recognized since the testcase is JUnit3 and not
> JUnit4:
> Solution:
> Migrate the testcase to JUnit4
> How:
> Remove extends TestCase"
> SetUp and TearDown methods
> @Override
> protected void setUp() throws Exception { }
> replaced by:
> @Before
> public void setUp() throws Exception { }
> Same for tearDown():
> @Override
> protected void tearDown() throws Exception { }
> replaced by
> @After
> public void tearDown() throws Exception { }
> Imports
> The imports has to be reorganized:
> Remove import junit.framework.TestCase;
> Add org.junit.*; or import org.junit.After; import org.junit.Before;
> import org.junit.Test;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4781) Unit test TestKerberosAuthenticationHandler fails with ant 1.8.3+
[
https://issues.apache.org/jira/browse/MAPREDUCE-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495490#comment-13495490
]
Amir Sanjar commented on MAPREDUCE-4781:
testcases do not fail on TRUNK because @ignore tag is not used, however it
should still be migrated to JUnit4, any suggestion?
> Unit test TestKerberosAuthenticationHandler fails with ant 1.8.3+
> -
>
> Key: MAPREDUCE-4781
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4781
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 1.0.3
> Environment: Fedora 17_64 on x86
>Reporter: Amir Sanjar
> Attachments: HADOOP-9024-branch-1.patch, MAPREDUCE-4781-branch-1.patch
>
>
> Problem:
> JUnit tag @Ignore is not recognized since the testcase is JUnit3 and not
> JUnit4:
> Solution:
> Migrate the testcase to JUnit4
> How:
> Remove extends TestCase"
> SetUp and TearDown methods
> @Override
> protected void setUp() throws Exception { }
> replaced by:
> @Before
> public void setUp() throws Exception { }
> Same for tearDown():
> @Override
> protected void tearDown() throws Exception { }
> replaced by
> @After
> public void tearDown() throws Exception { }
> Imports
> The imports has to be reorganized:
> Remove import junit.framework.TestCase;
> Add org.junit.*; or import org.junit.After; import org.junit.Before;
> import org.junit.Test;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariappan Asokan updated MAPREDUCE-2454: Status: Open (was: Patch Available) > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.2-alpha, 2.0.0-alpha, 3.0.0 >Reporter: Mariappan Asokan >Assignee: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, > KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, > mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz, > ReduceInputSorter.java > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4781) Unit test TestKerberosAuthenticationHandler fails with ant 1.8.3+
[
https://issues.apache.org/jira/browse/MAPREDUCE-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Sanjar updated MAPREDUCE-4781:
---
Attachment: HADOOP-9024-branch-1.patch
Patch for 1.0.3 and 1.0.4
> Unit test TestKerberosAuthenticationHandler fails with ant 1.8.3+
> -
>
> Key: MAPREDUCE-4781
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4781
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 1.0.3
> Environment: Fedora 17_64 on x86
>Reporter: Amir Sanjar
> Attachments: HADOOP-9024-branch-1.patch, MAPREDUCE-4781-branch-1.patch
>
>
> Problem:
> JUnit tag @Ignore is not recognized since the testcase is JUnit3 and not
> JUnit4:
> Solution:
> Migrate the testcase to JUnit4
> How:
> Remove extends TestCase"
> SetUp and TearDown methods
> @Override
> protected void setUp() throws Exception { }
> replaced by:
> @Before
> public void setUp() throws Exception { }
> Same for tearDown():
> @Override
> protected void tearDown() throws Exception { }
> replaced by
> @After
> public void tearDown() throws Exception { }
> Imports
> The imports has to be reorganized:
> Remove import junit.framework.TestCase;
> Add org.junit.*; or import org.junit.After; import org.junit.Before;
> import org.junit.Test;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4723) Fix warnings found by findbugs 2
[ https://issues.apache.org/jira/browse/MAPREDUCE-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495474#comment-13495474 ] Eli Collins commented on MAPREDUCE-4723: Thanks for posting the report Sandy. Went through all the changes, mostly the same things I found in HDFS-4014. +1 looks good > Fix warnings found by findbugs 2 > > > Key: MAPREDUCE-4723 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4723 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.0.0-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4723-1.patch, MAPREDUCE-4723.patch, > MAPREDUCE-4723.patch, MAPREDUCE-4723.patch, mapreduce-findbugs.tar.gz > > > The MAPREDUCE side of HADOOP-8594. Umbrella jira for fixing the warnings > found by findbugs 2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4425) Speculation + Fetch failures can lead to a hung job
[
https://issues.apache.org/jira/browse/MAPREDUCE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495458#comment-13495458
]
Hadoop QA commented on MAPREDUCE-4425:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553144/MAPREDUCE-4425-branch23.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3015//console
This message is automatically generated.
> Speculation + Fetch failures can lead to a hung job
> ---
>
> Key: MAPREDUCE-4425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Siddharth Seth
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: MAPREDUCE-4425-branch23.patch, MAPREDUCE-4425.patch,
> MAPREDUCE-4425.patch
>
>
> After a task goes to SUCCEEDED, FAILED/KILLED attempts are ignored.
> 1. attemp1 starts
> 2. speculative attempt starts
> 3. attempt 1 completes - Task moves to SUCCEEDED state
> 4. speculative attempt is KILLED
> 5. T_ATTEMPT_KILLED is ignored.
> 6. attemp1 1 fails with TOO_MANY_FETCH_FAILURES
> The job will effectively hang, since a new task attempt isn't started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4425) Speculation + Fetch failures can lead to a hung job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4425: -- Attachment: MAPREDUCE-4425-branch23.patch Thanks for looking at this, Bobby. Here's a patch for branch-0.23. The RetroactiveKilledAtSucceeded transition is missing in 0.23, so I preserved the existing 0.23 behavior if an attempt that already succeeded was killed. > Speculation + Fetch failures can lead to a hung job > --- > > Key: MAPREDUCE-4425 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4425 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Siddharth Seth >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-4425-branch23.patch, MAPREDUCE-4425.patch, > MAPREDUCE-4425.patch > > > After a task goes to SUCCEEDED, FAILED/KILLED attempts are ignored. > 1. attemp1 starts > 2. speculative attempt starts > 3. attempt 1 completes - Task moves to SUCCEEDED state > 4. speculative attempt is KILLED > 5. T_ATTEMPT_KILLED is ignored. > 6. attemp1 1 fails with TOO_MANY_FETCH_FAILURES > The job will effectively hang, since a new task attempt isn't started. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4425) Speculation + Fetch failures can lead to a hung job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495435#comment-13495435 ] Hudson commented on MAPREDUCE-4425: --- Integrated in Hadoop-trunk-Commit #3001 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3001/]) MAPREDUCE-4425. Speculation + Fetch failures can lead to a hung job (jlowe via bobby) (Revision 1408360) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408360 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java > Speculation + Fetch failures can lead to a hung job > --- > > Key: MAPREDUCE-4425 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4425 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Siddharth Seth >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-4425.patch, MAPREDUCE-4425.patch > > > After a task goes to SUCCEEDED, FAILED/KILLED attempts are ignored. > 1. attemp1 starts > 2. speculative attempt starts > 3. attempt 1 completes - Task moves to SUCCEEDED state > 4. speculative attempt is KILLED > 5. T_ATTEMPT_KILLED is ignored. > 6. attemp1 1 fails with TOO_MANY_FETCH_FAILURES > The job will effectively hang, since a new task attempt isn't started. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4425) Speculation + Fetch failures can lead to a hung job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495433#comment-13495433 ] Robert Joseph Evans commented on MAPREDUCE-4425: I put this into trunk, and branch-2, but it does not appear to be a clean merge to 0.23. Could you provide another patch for that? > Speculation + Fetch failures can lead to a hung job > --- > > Key: MAPREDUCE-4425 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4425 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Siddharth Seth >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-4425.patch, MAPREDUCE-4425.patch > > > After a task goes to SUCCEEDED, FAILED/KILLED attempts are ignored. > 1. attemp1 starts > 2. speculative attempt starts > 3. attempt 1 completes - Task moves to SUCCEEDED state > 4. speculative attempt is KILLED > 5. T_ATTEMPT_KILLED is ignored. > 6. attemp1 1 fails with TOO_MANY_FETCH_FAILURES > The job will effectively hang, since a new task attempt isn't started. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495428#comment-13495428 ] Hudson commented on MAPREDUCE-4787: --- Integrated in Hadoop-trunk-Commit #3000 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3000/]) MAPREDUCE-4787. TestJobMonitorAndPrint is broken (Rob Parker via bobby) (Revision 1408355) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408355 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml > TestJobMonitorAndPrint is broken > > > Key: MAPREDUCE-4787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Ravi Prakash >Assignee: Robert Parker > Labels: test > Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4787.patch > > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<< > FAILURE! > testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint) > Time elapsed: 1105 sec <<< ERROR! > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:243) > at junit.framework.TestSuite.run(TestSuite.java:238) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4425) Speculation + Fetch failures can lead to a hung job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495427#comment-13495427 ] Robert Joseph Evans commented on MAPREDUCE-4425: The patch looks fine to me. When a task fails or is killed retroactively we remove it from inProgess and add it to finished. Also when a task succeeds after already succeeding we add it to finished and remove it from inProgess. +1 I'll check it in. > Speculation + Fetch failures can lead to a hung job > --- > > Key: MAPREDUCE-4425 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4425 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Siddharth Seth >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-4425.patch, MAPREDUCE-4425.patch > > > After a task goes to SUCCEEDED, FAILED/KILLED attempts are ignored. > 1. attemp1 starts > 2. speculative attempt starts > 3. attempt 1 completes - Task moves to SUCCEEDED state > 4. speculative attempt is KILLED > 5. T_ATTEMPT_KILLED is ignored. > 6. attemp1 1 fails with TOO_MANY_FETCH_FAILURES > The job will effectively hang, since a new task attempt isn't started. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4787: --- Resolution: Fixed Fix Version/s: 0.23.5 2.0.3-alpha Status: Resolved (was: Patch Available) Thanks Rob, I put this into trunk, branch-2, and branch-0.23 > TestJobMonitorAndPrint is broken > > > Key: MAPREDUCE-4787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Ravi Prakash >Assignee: Robert Parker > Labels: test > Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4787.patch > > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<< > FAILURE! > testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint) > Time elapsed: 1105 sec <<< ERROR! > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:243) > at junit.framework.TestSuite.run(TestSuite.java:238) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495417#comment-13495417 ] Robert Joseph Evans commented on MAPREDUCE-4787: The change looks good to me. +1 pending Jenkins. > TestJobMonitorAndPrint is broken > > > Key: MAPREDUCE-4787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Ravi Prakash >Assignee: Robert Parker > Labels: test > Fix For: 3.0.0 > > Attachments: MAPREDUCE-4787.patch > > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<< > FAILURE! > testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint) > Time elapsed: 1105 sec <<< ERROR! > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:243) > at junit.framework.TestSuite.run(TestSuite.java:238) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495418#comment-13495418
]
Hadoop QA commented on MAPREDUCE-4787:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553135/MAPREDUCE-4787.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3014//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3014//console
This message is automatically generated.
> TestJobMonitorAndPrint is broken
>
>
> Key: MAPREDUCE-4787
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5
>Reporter: Ravi Prakash
>Assignee: Robert Parker
> Labels: test
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-4787.patch
>
>
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<<
> FAILURE!
> testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint)
> Time elapsed: 1105 sec <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4786) Job End Notification retry interval is 5 milliseconds by default
[
https://issues.apache.org/jira/browse/MAPREDUCE-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495413#comment-13495413
]
Robert Joseph Evans commented on MAPREDUCE-4786:
The change looks fine to me. My only comment is that I would prefer to see
mapreduce.job.end-notification.url have no default value in mapred-default.xml
instead of having it be commented out. When it is commented out then it will
not appear in the generated documentation for the file.
> Job End Notification retry interval is 5 milliseconds by default
>
>
> Key: MAPREDUCE-4786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4786
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.2-alpha, 0.23.4
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4786.branch-2.patch, MAPREDUCE-4786.patch
>
>
> Courtesy [~stevenwillis] and [~qwertymaniac]
> {quote}
> From: Harsh J
> I believe the configs of the latter of both of the above
> classifications were meant to be added in as replacement names, but
> the property names got added in wrong (as the former/older named ones)
> in the XML.
> the word "seconds" in the description of retries? The code in MR2's
> JobEndNotifier seems to expect seconds but uses it directly in
> Thread.sleep(…) without making it milliseconds, which may be a bug we
> need to fix as well, perhaps in a same issue as the configs ones.
> On Fri, Nov 9, 2012 at 11:21 PM, Steven Willis wrote:
> > And I noticed that there are some duplicate properties with different
> > values and different descriptions:
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4751) AM stuck in KILL_WAIT for days
[ https://issues.apache.org/jira/browse/MAPREDUCE-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4751: --- Resolution: Fixed Fix Version/s: 0.23.5 2.0.3-alpha 3.0.0 Status: Resolved (was: Patch Available) Thanks Vinod, I put this into trunk, branch-2, and branch-0.23 > AM stuck in KILL_WAIT for days > -- > > Key: MAPREDUCE-4751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3, 2.0.2-alpha >Reporter: Ravi Prakash >Assignee: Vinod Kumar Vavilapalli > Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4751-20121108.txt, > MAPREDUCE-4751-20121109.txt, MR-4751-branch-0.23.txt, > TaskAttemptStateGraph.jpg > > > We found some jobs were stuck in KILL_WAIT for days on end. The RM shows them > as RUNNING. When you go to the AM, it shows it in the KILL_WAIT state, and a > few maps running. All these maps were scheduled on nodes which are now in the > RM's Lost nodes list. The running maps are in the FAIL_CONTAINER_CLEANUP state -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-4787: - Attachment: MAPREDUCE-4787.patch > TestJobMonitorAndPrint is broken > > > Key: MAPREDUCE-4787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Ravi Prakash >Assignee: Robert Parker > Labels: test > Fix For: 3.0.0 > > Attachments: MAPREDUCE-4787.patch > > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<< > FAILURE! > testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint) > Time elapsed: 1105 sec <<< ERROR! > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:243) > at junit.framework.TestSuite.run(TestSuite.java:238) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4751) AM stuck in KILL_WAIT for days
[ https://issues.apache.org/jira/browse/MAPREDUCE-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495405#comment-13495405 ] Jason Lowe commented on MAPREDUCE-4751: --- The patch for branch-0.23 looks like a good port to me, +1. > AM stuck in KILL_WAIT for days > -- > > Key: MAPREDUCE-4751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3, 2.0.2-alpha >Reporter: Ravi Prakash >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-4751-20121108.txt, > MAPREDUCE-4751-20121109.txt, MR-4751-branch-0.23.txt, > TaskAttemptStateGraph.jpg > > > We found some jobs were stuck in KILL_WAIT for days on end. The RM shows them > as RUNNING. When you go to the AM, it shows it in the KILL_WAIT state, and a > few maps running. All these maps were scheduled on nodes which are now in the > RM's Lost nodes list. The running maps are in the FAIL_CONTAINER_CLEANUP state -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-4787: - Fix Version/s: 3.0.0 Status: Patch Available (was: In Progress) Changed pom file to use the existing test log4j.properties in hadoop-mapreduce-client-core > TestJobMonitorAndPrint is broken > > > Key: MAPREDUCE-4787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Ravi Prakash >Assignee: Robert Parker > Labels: test > Fix For: 3.0.0 > > Attachments: MAPREDUCE-4787.patch > > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<< > FAILURE! > testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint) > Time elapsed: 1105 sec <<< ERROR! > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:243) > at junit.framework.TestSuite.run(TestSuite.java:238) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work stopped] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-4787 stopped by Robert Parker. > TestJobMonitorAndPrint is broken > > > Key: MAPREDUCE-4787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Ravi Prakash >Assignee: Robert Parker > Labels: test > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<< > FAILURE! > testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint) > Time elapsed: 1105 sec <<< ERROR! > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:243) > at junit.framework.TestSuite.run(TestSuite.java:238) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-4787 started by Robert Parker. > TestJobMonitorAndPrint is broken > > > Key: MAPREDUCE-4787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Ravi Prakash >Assignee: Robert Parker > Labels: test > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<< > FAILURE! > testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint) > Time elapsed: 1105 sec <<< ERROR! > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:243) > at junit.framework.TestSuite.run(TestSuite.java:238) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-4787 started by Robert Parker. > TestJobMonitorAndPrint is broken > > > Key: MAPREDUCE-4787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Ravi Prakash >Assignee: Robert Parker > Labels: test > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<< > FAILURE! > testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint) > Time elapsed: 1105 sec <<< ERROR! > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:243) > at junit.framework.TestSuite.run(TestSuite.java:238) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker reassigned MAPREDUCE-4787: Assignee: Robert Parker > TestJobMonitorAndPrint is broken > > > Key: MAPREDUCE-4787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Ravi Prakash >Assignee: Robert Parker > Labels: test > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<< > FAILURE! > testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint) > Time elapsed: 1105 sec <<< ERROR! > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:243) > at junit.framework.TestSuite.run(TestSuite.java:238) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4789) Map Reduce Counters are 0 on the Job Tracker job details page
Jeff Lord created MAPREDUCE-4789: Summary: Map Reduce Counters are 0 on the Job Tracker job details page Key: MAPREDUCE-4789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4789 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 2.0.0-alpha Reporter: Jeff Lord Counters on the jobtracker details page are 0 for map and reduce columns. This is reproduced using a simple wordcount. Of note is that customer counters show properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4788) Job are marking as FAILED even if there are no failed tasks in it
Devaraj K created MAPREDUCE-4788: Summary: Job are marking as FAILED even if there are no failed tasks in it Key: MAPREDUCE-4788 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4788 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.2-alpha, 2.0.1-alpha Reporter: Devaraj K Assignee: Devaraj K Sometimes Jobs are marking as FAILED and some the tasks are marking as KILLED in it. In MRAppMaster, JobFinishEvent is triggering and waiting for the 5000 millis. If any tasks final state is unknown by this time those tasks are marking as KILLED and Job state is marking as FAILED. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495386#comment-13495386 ] Robert Joseph Evans commented on MAPREDUCE-4787: Looking at the logs for the failure it looks like the test is looking for a log4j file that was removed with the rest of the mrv1 code. > TestJobMonitorAndPrint is broken > > > Key: MAPREDUCE-4787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Ravi Prakash > Labels: test > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<< > FAILURE! > testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint) > Time elapsed: 1105 sec <<< ERROR! > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:243) > at junit.framework.TestSuite.run(TestSuite.java:238) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4751) AM stuck in KILL_WAIT for days
[
https://issues.apache.org/jira/browse/MAPREDUCE-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495380#comment-13495380
]
Hadoop QA commented on MAPREDUCE-4751:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553129/MR-4751-branch-0.23.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3013//console
This message is automatically generated.
> AM stuck in KILL_WAIT for days
> --
>
> Key: MAPREDUCE-4751
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4751
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.3, 2.0.2-alpha
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
> Attachments: MAPREDUCE-4751-20121108.txt,
> MAPREDUCE-4751-20121109.txt, MR-4751-branch-0.23.txt,
> TaskAttemptStateGraph.jpg
>
>
> We found some jobs were stuck in KILL_WAIT for days on end. The RM shows them
> as RUNNING. When you go to the AM, it shows it in the KILL_WAIT state, and a
> few maps running. All these maps were scheduled on nodes which are now in the
> RM's Lost nodes list. The running maps are in the FAIL_CONTAINER_CLEANUP state
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4751) AM stuck in KILL_WAIT for days
[ https://issues.apache.org/jira/browse/MAPREDUCE-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4751: --- Attachment: MR-4751-branch-0.23.txt I merged the patch to trunk, and branch-2 with no problems, but branch-0.23 has a few merge conflicts. I think I have resolved all of the conflicts correctly, but I would like another pair of eye to confirm that. I am running unit tests and so far they all look OK. > AM stuck in KILL_WAIT for days > -- > > Key: MAPREDUCE-4751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3, 2.0.2-alpha >Reporter: Ravi Prakash >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-4751-20121108.txt, > MAPREDUCE-4751-20121109.txt, MR-4751-branch-0.23.txt, > TaskAttemptStateGraph.jpg > > > We found some jobs were stuck in KILL_WAIT for days on end. The RM shows them > as RUNNING. When you go to the AM, it shows it in the KILL_WAIT state, and a > few maps running. All these maps were scheduled on nodes which are now in the > RM's Lost nodes list. The running maps are in the FAIL_CONTAINER_CLEANUP state -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4764) repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
[ https://issues.apache.org/jira/browse/MAPREDUCE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495376#comment-13495376 ] Daryn Sharp commented on MAPREDUCE-4764: Sure. Just be sure to also check that the tokens are in the current user's UGI. > repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile > > > Key: MAPREDUCE-4764 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4764 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ivan A. Veselovsky > Attachments: MAPREDUCE-4764-trunk.patch > > > the test is @Ignore-ed, and fails being enabled. > Suggested to repair it to fill the coverage gap. > Problems fixed in the test: > (1) MRConfig.FRAMEWORK_NAME and YarnConfiguration.RM_PRINCIPAL properties > must be correctly set in the configuration to correctly enable the security > in the way this test implies. > (2) The property MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY now is not > passed into the Job configuration -- it is intentionally deleted from there. > So, we pass the binary file name in another dedicated property. > (3) The test was using deprecated cluster classes. All them are updated to > the modern analogs. > (4) The delegation token found in the job context is now correctly compared > to the one deserialized from the binary file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4425) Speculation + Fetch failures can lead to a hung job
[
https://issues.apache.org/jira/browse/MAPREDUCE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495374#comment-13495374
]
Hadoop QA commented on MAPREDUCE-4425:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553125/MAPREDUCE-4425.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3012//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3012//console
This message is automatically generated.
> Speculation + Fetch failures can lead to a hung job
> ---
>
> Key: MAPREDUCE-4425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Siddharth Seth
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: MAPREDUCE-4425.patch, MAPREDUCE-4425.patch
>
>
> After a task goes to SUCCEEDED, FAILED/KILLED attempts are ignored.
> 1. attemp1 starts
> 2. speculative attempt starts
> 3. attempt 1 completes - Task moves to SUCCEEDED state
> 4. speculative attempt is KILLED
> 5. T_ATTEMPT_KILLED is ignored.
> 6. attemp1 1 fails with TOO_MANY_FETCH_FAILURES
> The job will effectively hang, since a new task attempt isn't started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4425) Speculation + Fetch failures can lead to a hung job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4425: -- Attachment: MAPREDUCE-4425.patch Upmerged patch to trunk since MAPREDUCE-4751 was integrated. > Speculation + Fetch failures can lead to a hung job > --- > > Key: MAPREDUCE-4425 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4425 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Siddharth Seth >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-4425.patch, MAPREDUCE-4425.patch > > > After a task goes to SUCCEEDED, FAILED/KILLED attempts are ignored. > 1. attemp1 starts > 2. speculative attempt starts > 3. attempt 1 completes - Task moves to SUCCEEDED state > 4. speculative attempt is KILLED > 5. T_ATTEMPT_KILLED is ignored. > 6. attemp1 1 fails with TOO_MANY_FETCH_FAILURES > The job will effectively hang, since a new task attempt isn't started. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4751) AM stuck in KILL_WAIT for days
[ https://issues.apache.org/jira/browse/MAPREDUCE-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495335#comment-13495335 ] Hudson commented on MAPREDUCE-4751: --- Integrated in Hadoop-trunk-Commit #2999 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/2999/]) MAPREDUCE-4751. AM stuck in KILL_WAIT for days (vinodkv via bobby) (Revision 1408314) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408314 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java > AM stuck in KILL_WAIT for days > -- > > Key: MAPREDUCE-4751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3, 2.0.2-alpha >Reporter: Ravi Prakash >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-4751-20121108.txt, > MAPREDUCE-4751-20121109.txt, TaskAttemptStateGraph.jpg > > > We found some jobs were stuck in KILL_WAIT for days on end. The RM shows them > as RUNNING. When you go to the AM, it shows it in the KILL_WAIT state, and a > few maps running. All these maps were scheduled on nodes which are now in the > RM's Lost nodes list. The running maps are in the FAIL_CONTAINER_CLEANUP state -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4751) AM stuck in KILL_WAIT for days
[ https://issues.apache.org/jira/browse/MAPREDUCE-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495321#comment-13495321 ] Robert Joseph Evans commented on MAPREDUCE-4751: The patch looks OK to me. It does pass all of the tests so I am +1 on it. We may be able to make things simpler like was stated, but for the time being I think this is OK. > AM stuck in KILL_WAIT for days > -- > > Key: MAPREDUCE-4751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3, 2.0.2-alpha >Reporter: Ravi Prakash >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-4751-20121108.txt, > MAPREDUCE-4751-20121109.txt, TaskAttemptStateGraph.jpg > > > We found some jobs were stuck in KILL_WAIT for days on end. The RM shows them > as RUNNING. When you go to the AM, it shows it in the KILL_WAIT state, and a > few maps running. All these maps were scheduled on nodes which are now in the > RM's Lost nodes list. The running maps are in the FAIL_CONTAINER_CLEANUP state -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4787) TestJobMonitorAndPrint is broken
Ravi Prakash created MAPREDUCE-4787: --- Summary: TestJobMonitorAndPrint is broken Key: MAPREDUCE-4787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4787 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 Reporter: Ravi Prakash Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.169 sec <<< FAILURE! testJobMonitorAndPrint(org.apache.hadoop.mapreduce.TestJobMonitorAndPrint) Time elapsed: 1105 sec <<< ERROR! java.lang.NullPointerException at org.apache.hadoop.mapreduce.TestJobMonitorAndPrint.testJobMonitorAndPrint(TestJobMonitorAndPrint.java:97) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4786) Job End Notification retry interval is 5 milliseconds by default
[
https://issues.apache.org/jira/browse/MAPREDUCE-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495316#comment-13495316
]
Ravi Prakash commented on MAPREDUCE-4786:
-
The test failure is unrelated to this patch. It fails without the patch as
well. I've filed https://issues.apache.org/jira/browse/MAPREDUCE-4787
> Job End Notification retry interval is 5 milliseconds by default
>
>
> Key: MAPREDUCE-4786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4786
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.2-alpha, 0.23.4
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4786.branch-2.patch, MAPREDUCE-4786.patch
>
>
> Courtesy [~stevenwillis] and [~qwertymaniac]
> {quote}
> From: Harsh J
> I believe the configs of the latter of both of the above
> classifications were meant to be added in as replacement names, but
> the property names got added in wrong (as the former/older named ones)
> in the XML.
> the word "seconds" in the description of retries? The code in MR2's
> JobEndNotifier seems to expect seconds but uses it directly in
> Thread.sleep(…) without making it milliseconds, which may be a bug we
> need to fix as well, perhaps in a same issue as the configs ones.
> On Fri, Nov 9, 2012 at 11:21 PM, Steven Willis wrote:
> > And I noticed that there are some duplicate properties with different
> > values and different descriptions:
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
