[
https://issues.apache.org/jira/browse/MAPREDUCE-6271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369996#comment-14369996
]
Gera Shegalov commented on MAPREDUCE-6271:
------------------------------------------
Cluster#getJob is a public API and used in different places such as 'mapred
job'. It redirects to JHS after the job is finished. If the job configured with
counter limits differently from the site/default settings it was not possible
to receive status of this job before MAPREDUCE-5875.
Part of the code that this JIRA restores
{code}
conf = new JobConf(status.getJobFile());
{code}
was also incorrect because it was a silent nop. This constructor loads job conf
from a staging directory as if it's a local file. However, that is only true in
the local mode. Normally it's on a global DFS.
I think having modifying the current code to avoid loading the defaults would
be a right fix JobConf(false) because job.xml already has them.
FileNotFoundException occurs due to a tolerable race condition when the files
are already moved to the history location but you still have not received an
updated status. In this case, we have no choice but fall back to the default
new JobConf().
> org.apache.hadoop.mapreduce.Cluster GetJob() display warn log
> -------------------------------------------------------------
>
> Key: MAPREDUCE-6271
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6271
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.7.0
> Reporter: Peng Zhang
> Assignee: Peng Zhang
> Attachments: MR-6271.patch
>
>
> When using getJob() with MapReduce 2.7, warn log caused by configuration
> loaded twice is displayed every time. And when job completed, this function
> will display warn log of "java.io.FileNotFoundException"
> And I think this is related with MAPREDUCE-5875, the change in GetJob() seems
> to be not needed, cause it's only for test.
> {noformat}
> 15/03/04 13:41:23 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 15/03/04 13:41:23 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 15/03/04 13:41:24 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 15/03/04 13:41:24 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 15/03/04 13:41:25 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 15/03/04 13:41:25 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 15/03/04 13:41:26 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 15/03/04 13:41:26 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 15/03/04 13:41:27 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 15/03/04 13:41:27 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 15/03/04 13:41:28 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 15/03/04 13:41:28 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 15/03/04 13:41:29 WARN conf.Configuration:
> hdfsG://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 15/03/04 13:41:29 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 15/03/04 13:41:29 INFO exec.Task: 2015-03-04 13:41:29,853 Stage-1 map = 100%,
> reduce = 0%, Cumulative CPU 2.37 sec
> 15/03/04 13:41:30 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 15/03/04 13:41:30 WARN conf.Configuration:
> hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> 15/03/04 13:41:31 WARN mapreduce.Cluster: Job conf missing on
> clusterjava.io.FileNotFoundException: File does not exist:
> /yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
