[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-1398: Attachment: mr-1398-y20.patch Updated patch for earlier version of Hadoop. Not for commit here. > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: mr-1398-y20.patch, patch-1398-1.txt, patch-1398-2.txt, > patch-1398-ydist.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Release Note: Fixed TaskLauncher to stop waiting for blocking slots, for a TIP that is killed / failed while it is in queue. > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1398-1.txt, patch-1398-2.txt, > patch-1398-ydist.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-1398: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thanks, Amareshwari ! > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1398-1.txt, patch-1398-2.txt, > patch-1398-ydist.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Attachment: patch-1398-ydist.txt Patch for Yahoo! distribution. Ran ant test and test-patch. test-patch failed because of MAPREDUCE-1497. All unit tests passed except TestNodeRefresh (due to MAPREDUCE-677). TestNodeRefresh passed when I reran the test. > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1398-1.txt, patch-1398-2.txt, > patch-1398-ydist.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Fix Version/s: 0.22.0 > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1398-1.txt, patch-1398-2.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Attachment: patch-1398-2.txt Patch with comments incorporated. bq. The default value for taskMemoryManagerEnabled was changed in the patch which seemed unnecessary. Can we instead override isTaskMemoryManagerEnabled, if we just want to short circuit this in the test case ? Instead of overriding isTaskMemoryManagerEnabled(), I made setTaskMemoryManagerEnabledFlag() method package private and called it from testcase to turn off memory management. > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1398-1.txt, patch-1398-2.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Status: Patch Available (was: Open) > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Attachments: patch-1398-1.txt, patch-1398-2.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Status: Open (was: Patch Available) > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Attachments: patch-1398-1.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Attachment: patch-1398-1.txt Added one more assertion to the testcase. > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Attachments: patch-1398-1.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Assignee: Amareshwari Sriramadasu Status: Patch Available (was: Open) > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Attachments: patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Attachment: patch-1398.txt Patch fixing the bug. Added a testcase which fails without the patch and passes with the patch. > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala > Attachments: patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
