[jira] [Commented] (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016996#comment-13016996 ] Hudson commented on MAPREDUCE-1974: --- Integrated in Hadoop-Mapreduce-trunk #643 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/]) > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993929#comment-12993929 ] Hudson commented on MAPREDUCE-1974: --- Integrated in Hadoop-Mapreduce-22-branch #33 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/]) > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990874#comment-12990874 ] Scott Chen commented on MAPREDUCE-1974: --- Thanks for the help, guys :) > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990827#comment-12990827 ] Hudson commented on MAPREDUCE-1974: --- Integrated in Hadoop-Mapreduce-trunk-Commit #603 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/603/]) MAPREDUCE-1974. Fix multiple preemtions of the same task in FairScheduler. Contributed by Scott Chen. > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990775#comment-12990775 ] Priyo Mustafi commented on MAPREDUCE-1974: -- I have tested this as well and all looks good > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990368#comment-12990368 ] Scott Chen commented on MAPREDUCE-1974: --- test-patch result [exec] [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990359#comment-12990359 ] Scott Chen commented on MAPREDUCE-1974: --- I posted the result on the wrong patch. It should not have a +1 on test included. Will run this again. > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[
https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990356#comment-12990356
]
Scott Chen commented on MAPREDUCE-1974:
---
{code}
[exec]
[exec] BUILD SUCCESSFUL
[exec] Total time: 2 minutes 15 seconds
[exec]
[exec]
[exec]
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test
framework compile.
[exec]
[exec]
[exec]
[exec]
[exec]
==
[exec]
==
[exec] Finished build.
[exec]
==
[exec]
==
{code}
> FairScheduler can preempt the same task many times
> --
>
> Key: MAPREDUCE-1974
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
>Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0
>Reporter: Scott Chen
>Assignee: Scott Chen
> Attachments: MAPREDUCE-1974.txt
>
>
> In FairScheduler.preemptTasks(), tasks are collected from
> JobInProgress.runningMapCache.
> But tasks repeat multiple times in JobInProgress.runningMapCache (on rack,
> node and cluster).
> This makes FairScheduler preempt the same task many times.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990329#comment-12990329 ] Joydeep Sen Sarma commented on MAPREDUCE-1974: -- +1. this seems like a no-brainer! > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12964584#action_12964584 ] Scott Chen commented on MAPREDUCE-1974: --- Hey Matei, Sorry for not responding on this for a while. We have put this in production for a long while and it fixes the problem. I think it is OK to commit this. > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12964479#action_12964479 ] Matei Zaharia commented on MAPREDUCE-1974: -- Did you guys fix this one at Facebook? It would be nice to put it into trunk and 0.21.1 too. > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903099#action_12903099 ] Scott Chen commented on MAPREDUCE-1974: --- Hi Matei, I see multiple same preemption in the log. I think I should add a test here to verify that. Scott > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903055#action_12903055 ] Matei Zaharia commented on MAPREDUCE-1974: -- The patch looks good to me. Is there any way to test this? (Did you at least try to print the list before and see that a task is in there multiple times?) > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
