[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878659#action_12878659 ] Scott Chen commented on MAPREDUCE-1829: --- Thanks for your help, Vinod and Ravi :) > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829-20100610.txt, MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878241#action_12878241 ] Vinod K V commented on MAPREDUCE-1829: -- bq. -1 tests included. The patch doesn't appear to include any new or modified tests. This is a performance improvement. The correctness of the code in question is already tested by TestSpeculativeExecution.testTaskLATEScheduling(). > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829-20100610.txt, MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878239#action_12878239 ] Vinod K V commented on MAPREDUCE-1829: -- TestCopyFiles and TestSimulatorDeterministicReplay are usual suspects - MAPREDUCE-1858 and MAPREDUCE-1834. I am going to check this in. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829-20100610.txt, MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878187#action_12878187 ] Hadoop QA commented on MAPREDUCE-1829: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446753/MAPREDUCE-1829-20100610.txt against trunk revision 953879. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/237/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/237/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/237/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/237/console This message is automatically generated. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829-20100610.txt, MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877969#action_12877969 ] Scott Chen commented on MAPREDUCE-1829: --- I was thinking of "latest" because we are using LateComparator which compares the finish time. But as you explained, I think "slowest" is also a good name. +1 > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829-20100610.txt, MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877952#action_12877952 ] Hadoop QA commented on MAPREDUCE-1829: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446753/MAPREDUCE-1829-20100610.txt against trunk revision 953670. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/236/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/236/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/236/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/236/console This message is automatically generated. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829-20100610.txt, MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877816#action_12877816 ] Hadoop QA commented on MAPREDUCE-1829: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446753/MAPREDUCE-1829-20100610.txt against trunk revision 953490. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/563/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/563/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/563/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/563/console This message is automatically generated. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829-20100610.txt, MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877719#action_12877719 ] Ravi Gummadi commented on MAPREDUCE-1829: - I think slowestTIP is more meaningful because we are launching speculative attempt for the slowest task only. No ? Why do we call the tip selected(in this method) as "latestTIP" ? > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829-20100610.txt, MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877510#action_12877510 ] Scott Chen commented on MAPREDUCE-1829: --- Cool. The code looks neater this way. One small comment: Should we use "latestTIP" instead of "slowestTIP" to make it more clear? > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829-20100610.txt, MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876954#action_12876954 ] Ravi Gummadi commented on MAPREDUCE-1829: - Patch looks good. +1 > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876953#action_12876953 ] Ravi Gummadi commented on MAPREDUCE-1829: - TestSimulatorDeterministicReplay timeout is a known issue in trunk reported at MAPREDUCE-1834. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[
https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876831#action_12876831
]
Scott Chen commented on MAPREDUCE-1829:
---
Run the failed contrib test on my box again. It succeed. And I don't think this
is related to the change.
{code}
Testsuite: org.apache.hadoop.mapred.TestSimulatorDeterministicReplay
Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 35.856 sec
- Standard Output ---
Job job_200904211745_0002 is submitted at 103010
Job job_200904211745_0002 completed at 141990 with status: SUCCEEDED runtime:
38980
Job job_200904211745_0003 is submitted at 984078
Job job_200904211745_0004 is submitted at 993516
Job job_200904211745_0003 completed at 1011051 with status: SUCCEEDED runtime:
26973
Job job_200904211745_0005 is submitted at 1033963
Done, total events processed: 595469
Job job_200904211745_0002 is submitted at 103010
Job job_200904211745_0002 completed at 141990 with status: SUCCEEDED runtime:
38980
Job job_200904211745_0003 is submitted at 984078
Job job_200904211745_0004 is submitted at 993516
Job job_200904211745_0003 completed at 1011051 with status: SUCCEEDED runtime:
26973
Job job_200904211745_0005 is submitted at 1033963
Done, total events processed: 595469
- ---
{code}
> JobInProgress.findSpeculativeTask should use min() to find the candidate
> instead of sort()
> --
>
> Key: MAPREDUCE-1829
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Affects Versions: 0.22.0
>Reporter: Scott Chen
>Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1829.txt
>
>
> findSpeculativeTask needs only one candidate to speculate so it does not need
> to sort the whole list. It may looks OK but someone can still submit big jobs
> with small slow task thresholds. In this case, this sorting becomes expensive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876816#action_12876816 ] Hadoop QA commented on MAPREDUCE-1829: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446108/MAPREDUCE-1829.txt against trunk revision 952548. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/230/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/230/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/230/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/230/console This message is automatically generated. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[
https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876815#action_12876815
]
Scott Chen commented on MAPREDUCE-1829:
---
I have run "ant clean test" on my dev box. All tests passed.
{code}
test:
BUILD SUCCESSFUL
Total time: 186 minutes 23 seconds
{code}
> JobInProgress.findSpeculativeTask should use min() to find the candidate
> instead of sort()
> --
>
> Key: MAPREDUCE-1829
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Affects Versions: 0.22.0
>Reporter: Scott Chen
>Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1829.txt
>
>
> findSpeculativeTask needs only one candidate to speculate so it does not need
> to sort the whole list. It may looks OK but someone can still submit big jobs
> with small slow task thresholds. In this case, this sorting becomes expensive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[
https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876742#action_12876742
]
Scott Chen commented on MAPREDUCE-1829:
---
The tests failed because of class not found exception.
For example, java.lang.NoClassDefFoundError:
org/apache/hadoop/security/RefreshUserToGroupMappingsProtocol
I do not know why this happened.
I have manually run the test, TestSpeculativeExecution which covers this code
path and it succeed.
{code}
Testsuite: org.apache.hadoop.mapred.TestSpeculativeExecution
Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 39.893 sec
{code}
There is no unit test included because the code path is been tested by the
existing TestSpeculativeExecution.
I am submitting this to Hudson again.
> JobInProgress.findSpeculativeTask should use min() to find the candidate
> instead of sort()
> --
>
> Key: MAPREDUCE-1829
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Affects Versions: 0.22.0
>Reporter: Scott Chen
>Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1829.txt
>
>
> findSpeculativeTask needs only one candidate to speculate so it does not need
> to sort the whole list. It may looks OK but someone can still submit big jobs
> with small slow task thresholds. In this case, this sorting becomes expensive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875835#action_12875835 ] Hadoop QA commented on MAPREDUCE-1829: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446108/MAPREDUCE-1829.txt against trunk revision 951486. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/223/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/223/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/223/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/223/console This message is automatically generated. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874684#action_12874684 ] Scott Chen commented on MAPREDUCE-1829: --- Thanks Ravi. I am glad to help. I will figure out why contrib tests failed. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874558#action_12874558 ] Hadoop QA commented on MAPREDUCE-1829: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446108/MAPREDUCE-1829.txt against trunk revision 950286. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/554/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/554/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/554/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/554/console This message is automatically generated. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874312#action_12874312 ] Hadoop QA commented on MAPREDUCE-1829: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446055/MAPREDUCE-1829.txt against trunk revision 950021. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/552/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/552/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/552/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/552/console This message is automatically generated. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874157#action_12874157 ] Hadoop QA commented on MAPREDUCE-1829: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446040/MAPREDUCE-1829.txt against trunk revision 950021. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/551/console This message is automatically generated. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874141#action_12874141 ] Hadoop QA commented on MAPREDUCE-1829: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446037/MAPREDUCE-1829.txt against trunk revision 950021. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/550/console This message is automatically generated. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874138#action_12874138 ] Hadoop QA commented on MAPREDUCE-1829: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446037/MAPREDUCE-1829.txt against trunk revision 950021. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/217/console This message is automatically generated. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1829) JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874132#action_12874132 ] Scott Chen commented on MAPREDUCE-1829: --- There is no unit test change because the code path is already tested by the original unit tests. > JobInProgress.findSpeculativeTask should use min() to find the candidate > instead of sort() > -- > > Key: MAPREDUCE-1829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1829.txt > > > findSpeculativeTask needs only one candidate to speculate so it does not need > to sort the whole list. It may looks OK but someone can still submit big jobs > with small slow task thresholds. In this case, this sorting becomes expensive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
