[jira] [Commented] (MAPREDUCE-6185) YARN M/R may return NaN for reducer progress after Job completion

[sam liu (JIRA)]

    [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14237635#comment-14237635
 ] 

sam liu commented on MAPREDUCE-6185:
------------------------------------

This issue is about method 'jobClient.getJob(jobid).reduceProgress()' and only 
happens on the job which only has map tasks, but no reduce task, such as 
Teragen.

[A] My Test Code:
// Using hadoop-mapreduce-client-core\org.apache.hadoop.mapred.JobClient.java
                JobClient jobClient = new JobClient(new Configuration()); 
                
                for (int i = 0; i < 120; i++){
                        Thread.sleep(1000);
                        System.out.println("jobClient.getJob(" + args[0] + 
").reduceProgress()=" + jobClient.getJob(args[0]).reduceProgress());            
     
                }

[B] WordCount: Got expected result
Can get reduceProgress during job running(0.0 to 1.0) and after job 
completion(1.0).

$ hadoop jar yarn.hadoop.jar hadoop.yarn.proj.test.YarnClientTest 
job_1417679271770_0169
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.20833334
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.20833334
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.20833334
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.33333334
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.33333334
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.33333334
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.6765683
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.6765683
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.6765683
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.76213264
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.76213264
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.76213264
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.8284452
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.8284452
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.8284452
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.9177592
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.9177592
jobClient.getJob(job_1417679271770_0169).reduceProgress()=0.9177592
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
14/12/04 22:54:21 INFO mapred.ClientServiceDelegate: Application state is 
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0
jobClient.getJob(job_1417679271770_0169).reduceProgress()=1.0

[C] Teragen: Failed to get expected result
- Can get reduceProgress during job running(0.0 ) 
- But get NaN after job completion

As Teragen only has Mapper, but does not has Reducer, so I think it's correct.

$ hadoop jar yarn.hadoop.jar hadoop.yarn.proj.test.YarnClientTest 
job_1417679271770_0168
14/12/04 22:51:53 INFO client.RMProxy: Connecting to ResourceManager at 
cluster.nn1/9.31.111.201:8032
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
jobClient.getJob(job_1417679271770_0168).reduceProgress()=0.0
14/12/04 22:52:17 INFO mapred.ClientServiceDelegate: Application state is 
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
jobClient.getJob(job_1417679271770_0168).reduceProgress()=NaN
jobClient.getJob(job_1417679271770_0168).reduceProgress()=NaN
jobClient.getJob(job_1417679271770_0168).reduceProgress()=NaN
jobClient.getJob(job_1417679271770_0168).reduceProgress()=NaN
jobClient.getJob(job_1417679271770_0168).reduceProgress()=NaN

Expected result: '0.0'.
Actual result:  'NaN'.
Comment:
- Even after job completion, method reduceProgress() still should return same 
result '0.0', but not 'NaN'.
- As 'Redirecting to job history server', marked the component of this jira as 
'jobhistoryserver'

> YARN M/R may return NaN for reducer progress after Job completion
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-6185
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6185
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>    Affects Versions: 2.4.1
>            Reporter: sam liu
>            Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)