[jira] [Commented] (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016902#comment-13016902 ] Hudson commented on MAPREDUCE-1783: --- Integrated in Hadoop-Mapreduce-trunk #643 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/]) > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Fix For: 0.22.0, 0.23.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch, > submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008253#comment-13008253 ] Hudson commented on MAPREDUCE-1783: --- Integrated in Hadoop-Mapreduce-22-branch #38 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/38/]) > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Fix For: 0.22.0, 0.23.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch, > submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995004#comment-12995004 ] Konstantin Boudnik commented on MAPREDUCE-1783: --- I have fixed it. > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Fix For: 0.22.0, 0.23.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch, > submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995003#comment-12995003 ] Konstantin Boudnik commented on MAPREDUCE-1783: --- Looks like when the change has been committed to 0.22 the CHANGE.txt file was updated improperly (added to IMPROVEMENT section instead of BUG FIXES) which cases problems now for downstream merges. Also, the description of the JIRA has been written to CHANGES.txt differently from what it say on the ticket ;( Please fix. > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Fix For: 0.22.0, 0.23.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch, > submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993927#comment-12993927 ] Hudson commented on MAPREDUCE-1783: --- Integrated in Hadoop-Mapreduce-22-branch #33 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/]) > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Fix For: 0.22.0, 0.23.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch, > submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989454#comment-12989454 ] Scott Chen commented on MAPREDUCE-1783: --- I just committed this to 0.22. > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Fix For: 0.22.0, 0.23.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch, > submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988987#comment-12988987 ] Priyo Mustafi commented on MAPREDUCE-1783: -- Hi Scott, Can you commit this on 0.22 as well? Thanks > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Fix For: 0.23.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch, > submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966834#action_12966834 ] Hudson commented on MAPREDUCE-1783: --- Integrated in Hadoop-Mapreduce-trunk-Commit #557 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/557/]) MAPREDUCE-1783. FairScheduler initializes tasks only when the job can be run. (Ramkumar Vadali via schen) > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Fix For: 0.23.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch, > submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12964921#action_12964921 ] Scott Chen commented on MAPREDUCE-1783: --- +1 The patch looks good. And we have been running this in our production cluster for a while. I will commit this later. > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Fix For: 0.22.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch, > submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[
https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934065#action_12934065
]
Ramkumar Vadali commented on MAPREDUCE-1783:
Latest patch TEST RESULTS:
One test fails, but that also fails on a clean checkout
{code}
[junit] Test org.apache.hadoop.mapred.TestControlledMapReduceJob FAILED
(timeout)
{code}
ant test-patch succeeds:
{code}
[exec]
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test
framework compile.
[exec]
[exec]
[exec]
[exec]
[exec]
==
[exec]
==
[exec] Finished build.
[exec]
==
[exec]
==
[exec]
[exec]
BUILD SUCCESSFUL
Total time: 13 minutes 6 seconds
Test results are in /tmp/rvadali.hadoopQA
{code}
> Task Initialization should be delayed till when a job can be run
>
>
> Key: MAPREDUCE-1783
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
>Affects Versions: 0.20.1
>Reporter: Ramkumar Vadali
>Assignee: Ramkumar Vadali
> Fix For: 0.22.0
>
> Attachments: 0001-Pool-aware-job-initialization.patch,
> 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch,
> submit-mapreduce-1783.patch
>
>
> The FairScheduler task scheduler uses PoolManager to impose limits on the
> number of jobs that can be running at a given time. However, jobs that are
> submitted are initiaiized immediately by EagerTaskInitializationListener by
> calling JobInProgress.initTasks. This causes the job split file to be read
> into memory. The split information is not needed until the number of running
> jobs is less than the maximum specified. If the amount of split information
> is large, this leads to unnecessary memory pressure on the Job Tracker.
> To ease memory pressure, FairScheduler can use another implementation of
> JobInProgressListener that is aware of PoolManager limits and can delay task
> initialization until the number of running jobs is below the maximum.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884367#action_12884367 ] Hadoop QA commented on MAPREDUCE-1783: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445178/submit-mapreduce-1783.patch against trunk revision 959509. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/279/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/279/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/279/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/279/console This message is automatically generated. > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali > Fix For: 0.22.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870141#action_12870141 ] Hadoop QA commented on MAPREDUCE-1783: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445178/submit-mapreduce-1783.patch against trunk revision 946955. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/543/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/543/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/543/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/543/console This message is automatically generated. > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali > Fix For: 0.22.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869832#action_12869832 ] Hadoop QA commented on MAPREDUCE-1783: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445113/0001-Pool-aware-job-initialization.patch.1 against trunk revision 946833. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/541/console This message is automatically generated. > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali > Fix For: 0.22.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1 > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869821#action_12869821 ] Scott Chen commented on MAPREDUCE-1783: --- +1 The patch looks good to me. > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1 > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869782#action_12869782 ] Scott Chen commented on MAPREDUCE-1783: --- initializing should not be set back to false otherwise there may be a race condition. > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali > Attachments: 0001-Pool-aware-job-initialization.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
