[
https://issues.apache.org/jira/browse/MAPREDUCE-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171728#comment-14171728
]
Gera Shegalov commented on MAPREDUCE-4818:
------------------------------------------
Thanks for the patch, [~l201514]. Some comments:
nits:
AllocateResponsePBImpl: check indentation on L761-765
ContainerStatusPBImpl: Whitespace-only change L86-189
org.apache.hadoop.yarn.server.utils.BuilderUtils: Add new line on 194
ContainerImpl:
L1032: I think if totalNum is 0, the the progress should be 1.0
RMNodeImpl: there is an anti-pattern of double hash table lookups:
{code}
if (map.contains(key)) {
val = map.get(ley)
{code}
replace everywhere with
{code}
val = map.get(key)
if (val == null) {
// key not found
} else {
// key found
}
{code}
Right now isContainersInProgress is a parallel map to {{launchedContainers}}.
Can we simply add a boolean to ContainerStatus for a more efficient and compact
representation?
CapacityScheduler: Did you notice build failure because of bad merge. '}' is
missing on L1485
But this change does not even seem necessary. The implementation is the same
for all schedulers and can be lifted to AbstractYarnScheduler.
Do we have a test demonstrating that container stuck in localization still
times out?
There is a substantial work in YARN that should be a YARN JIRA. I leave it to
committers whether we need to make MR changes as a separate JIRA.
> Easier identification of tasks that timeout during localization
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-4818
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4818
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
> Affects Versions: 0.23.3, 2.0.3-alpha
> Reporter: Jason Lowe
> Assignee: Siqi Li
> Labels: usability
> Attachments: MAPREDUCE-4818.v1.patch, MAPREDUCE-4818.v2.patch,
> MAPREDUCE-4818.v3.patch, MAPREDUCE-4818.v4.patch
>
>
> When a task is taking too long to localize and is killed by the AM due to
> task timeout, the job UI/history is not very helpful. The attempt simply
> lists a diagnostic stating it was killed due to timeout, but there are no
> logs for the attempt since it never actually got started. There are log
> messages on the NM that show the container never made it past localization by
> the time it was killed, but users often do not have access to those logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
