[
https://issues.apache.org/jira/browse/MAPREDUCE-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467935#comment-13467935
]
Robert Joseph Evans commented on MAPREDUCE-3936:
------------------------------------------------
Under YARN the History Server also reads in the counters and that is where the
counter limits are needed in some form or another. I don't really know what
the correct solution is though. The history server cannot rely on the AM to
enforce a limit because it is user code, so if a job goes over the counter
limits what should the history server do? Especially because the client may be
requesting data from the history server after the AM exits, so we really want
the history server to return results as close to what the AM did as possible.
Should the history server refuse to load the job and throw an exception? That
would result in the client possibly failing, unless we are sure to update the
client to deal with that eventuality (I think this is the current behavior).
Should it just not load all of the counters? That would result in odd behavior
where the client would get a different view of the results when talking to the
history server instead of talking to the AM, but it wouldn't crash.
> Clients should not enforce counter limits
> ------------------------------------------
>
> Key: MAPREDUCE-3936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-3936.patch, MAPREDUCE-3936.patch
>
>
> The code for enforcing counter limits (from MAPREDUCE-1943) creates a static
> JobConf instance to load the limits, which may throw an exception if the
> client limit is set to be lower than the limit on the cluster (perhaps
> because the cluster limit was raised from the default).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
