[
https://issues.apache.org/jira/browse/MAPREDUCE-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838089#action_12838089
]
Scott Chen commented on MAPREDUCE-1221:
---------------------------------------
@Arun: I agree. It is reasonable to track these two.
We have tried once with the per-task limit. We tried to set the threshold
higher but it cannot be too high because it hurts the memory utilization. But
that threshold still cause some of our pipeline and user jobs fails. So we
switched it off quickly. But we really need some memory protecting feature.
That was why we started working on this patch.
Correct me if I am wrong. Kill a task will not make the job fail. It is
different from failing a task. So the 4th attempt gets killed will not fail the
job. Also, if it gets killed, it is likely that itself is the rough task
because it uses the highest amount of memory.
But in general, I agree with you. The
total-RSS-memory-based-and-killing-highest-memory-task approach is not as
predictable as the virtual-memory-limit approach. But it still have some
benefits that we discussed above. And it is optional to use. This patch will
not do anything if we don't configure it.
Again, thank you for the comment. I think the discussions are very helpful.
> Kill tasks on a node if the free physical memory on that machine falls below
> a configured threshold
> ---------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1221
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1221
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1221-v1.patch, MAPREDUCE-1221-v2.patch,
> MAPREDUCE-1221-v3.patch
>
>
> The TaskTracker currently supports killing tasks if the virtual memory of a
> task exceeds a set of configured thresholds. I would like to extend this
> feature to enable killing tasks if the physical memory used by that task
> exceeds a certain threshold.
> On a certain operating system (guess?), if user space processes start using
> lots of memory, the machine hangs and dies quickly. This means that we would
> like to prevent map-reduce jobs from triggering this condition. From my
> understanding, the killing-based-on-virtual-memory-limits (HADOOP-5883) were
> designed to address this problem. This works well when most map-reduce jobs
> are Java jobs and have well-defined -Xmx parameters that specify the max
> virtual memory for each task. On the other hand, if each task forks off
> mappers/reducers written in other languages (python/php, etc), the total
> virtual memory usage of the process-subtree varies greatly. In these cases,
> it is better to use kill-tasks-using-physical-memory-limits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
