[jira] [Commented] (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1367#comment-1367 ] Johannes Zillmann commented on MAPREDUCE-1367: -- Hey guys, is 0.21.0 the only branch this is fixed in ? Not in 1.x, not in 2.x ? > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367-branch1.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13547161#comment-13547161 ] Sandy Ryza commented on MAPREDUCE-1367: --- TestLocalRunner passes > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367-branch1.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13547159#comment-13547159 ] Sandy Ryza commented on MAPREDUCE-1367: --- Regenerated branch-1 patch > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367-branch1.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471916#comment-13471916 ] Sandy Ryza commented on MAPREDUCE-1367: --- I submitted a patch for branch-1, as it was never backported. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367-branch1.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965054#action_12965054 ] Ramkumar Vadali commented on MAPREDUCE-1367: @Aaron, Just curious, is this being used in production? If so, could you please outline the use case? > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828467#action_12828467 ] Hudson commented on MAPREDUCE-1367: --- Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/]) > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806693#action_12806693 ] Hudson commented on MAPREDUCE-1367: --- Integrated in Hadoop-Mapreduce-trunk #221 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/]) . LocalJobRunner should support parallel mapper execution. Contributed by Aaron Kimball. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806590#action_12806590 ] Aaron Kimball commented on MAPREDUCE-1367: -- Test failures seem unrelated - classpath issues on hudson again? > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806585#action_12806585 ] Hadoop QA commented on MAPREDUCE-1367: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431823/MAPREDUCE-1367.7.patch against trunk revision 904609. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/console This message is automatically generated. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.7.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805390#action_12805390 ] Aaron Kimball commented on MAPREDUCE-1367: -- Test failures are all unrelated (TT blacklisting has now falied two tests in a row?) > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805389#action_12805389 ] Hadoop QA commented on MAPREDUCE-1367: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431509/MAPREDUCE-1367.6.patch against trunk revision 903508. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/console This message is automatically generated. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.6.patch, > MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805302#action_12805302 ] Tom White commented on MAPREDUCE-1367: -- Aaron, this patch has gone stale. Could you regenerate, then I think it's ready to be committed. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802036#action_12802036 ] Jeff Hammerbacher commented on MAPREDUCE-1367: -- What is the delay in committing this patch? > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799570#action_12799570 ] Hadoop QA commented on MAPREDUCE-1367: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12430081/MAPREDUCE-1367.5.patch against trunk revision 898486. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/console This message is automatically generated. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.5.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799484#action_12799484 ] Hadoop QA commented on MAPREDUCE-1367: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12430043/MAPREDUCE-1367.4.patch against trunk revision 898486. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/console This message is automatically generated. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799448#action_12799448 ] Todd Lipcon commented on MAPREDUCE-1367: - MapTaskRunnable: make most of the instance variables final? - it should be clearer that mapOutputFiles is an "out param" for getMapTaskRunnables - ie that it's expected to be intitially empty. Alternatively, it looks like you may be able to remove this variable entirely by instead storing the MapOutputFile in the MapRunnable instance, and then iterating directly over the MapRunnables in the reducer. Does that make sense? However, the way you've done it is less invasive to the reduce side, so if you don't see the benefit, feel free to ignore this suggestion. - Does this handle the degenerate case of 0-map jobs? It sounds ridiculous, but I recall previous JIRAs for this situation, since occasionally people have a cron job that periodically processes a given directory. If the directory is empty, it may generate a job with no input splits and thus no tasks. Aside from that, looks good to me. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.3.patch, > MAPREDUCE-1367.4.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799069#action_12799069 ] Aaron Kimball commented on MAPREDUCE-1367: -- Tests that failed in Hudson all pass locally. These tests don't appear to use the LocalJobRunner. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799042#action_12799042 ] Hadoop QA commented on MAPREDUCE-1367: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429958/MAPREDUCE-1367.2.patch against trunk revision 898019. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/console This message is automatically generated. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.2.patch, MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1367) LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798299#action_12798299 ] Hadoop QA commented on MAPREDUCE-1367: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429800/MAPREDUCE-1367.patch against trunk revision 897118. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/console This message is automatically generated. > LocalJobRunner should support parallel mapper execution > --- > > Key: MAPREDUCE-1367 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1367 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1367.patch > > > The LocalJobRunner currently supports only a single execution thread. Given > the prevalence of multi-core CPUs, it makes sense to allow users to run > multiple tasks in parallel for improved performance on small (local-only) > jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
