[
https://issues.apache.org/jira/browse/MAPREDUCE-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950551#comment-14950551
]
Jason Lowe commented on MAPREDUCE-6489:
---------------------------------------
Thanks for the patch, Maysam!
I think it would be a little easier for users if they could configure zero or a
negative number in the limit to disable it rather than a giant value. The
description of the property should explain that this limit only applies to
writes that go through the Hadoop filesystem APIs within the task process
(i.e.: writes that will update the local filesystem BYTES_WRITTEN counter). It
does not cover other writes such as logging, sideband writes from subprocesses
(e.g.: streaming jobs), etc.
Should we be using exit code 65? I got the impression the original code was
trying to use different exit codes for different task failure reasons. Seems
like this would deserve a separate code.
The warn message should say why the task is being killed, otherwise users will
have little clue if the INFO message is suppressed. Speaking of users, this
doesn't fail in a very graceful way for users to determine what happened. The
history will just show the task exiting with exit code 65 (or some other
number), with no useful diagnostic message sent to the AM explaining what went
wrong. We should use umbilical.fatalError to report the fatal error before
tearing down so the UI and history has a useful diagnostic that clearly shows
why the task failed.
TASK_LOCAL_WRITE_LIMIT and DEFAULT_TASK_LOCAL_WRITE_LIMIT should be public
static final.
Nit: write.limit.bytes should be write-limit-bytes to be consistent with
local-fs, otherwise one would expect local.fs.write.limit.bytes. Normally dots
separate namespaces and dashes take the place of spaces for an identifier
within the namespace. Granted the existing code is very inconsistent about
this.
It would be nice to see a test that verifies writes to the local filesystem
trigger the failure. As it is now the test would pass even if we were looking
at the wrong counter or for some reason the counter wasn't working properly.
> Fail fast rogue tasks that write too much to local disk
> -------------------------------------------------------
>
> Key: MAPREDUCE-6489
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6489
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 2.7.1
> Reporter: Maysam Yabandeh
> Assignee: Maysam Yabandeh
> Attachments: MAPREDUCE-6489.001.patch, MAPREDUCE-6489.002.patch
>
>
> Tasks of the rogue jobs can write too much to local disk, negatively
> affecting the jobs running in collocated containers. Ideally YARN will be
> able to limit amount of local disk used by each task: YARN-4011. Until then,
> the mapreduce task can fail fast if the task is writing too much (above a
> configured threshold) to local disk.
> As we discussed
> [here|https://issues.apache.org/jira/browse/YARN-4011?focusedCommentId=14902750&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14902750]
> the suggested approach is that the MapReduce task checks for BYTES_WRITTEN
> counter for the local disk and throws an exception when it goes beyond a
> configured value. It is true that written bytes is larger than the actual
> used disk space, but to detect a rogue task the exact value is not required
> and a very large value for written bytes to local disk is a good indicative
> that the task is misbehaving.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)