[
https://issues.apache.org/jira/browse/PIG-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988300#action_12988300
]
Arun C Murthy commented on PIG-1829:
------------------------------------
Reading directly from JobHistory directory is a non-starter - permissions.
We should investigate standard apis. For now getting stats right after job
completion will work pretty much all the time. The JT has a the ability to be
stats for completed jobs either from memory or disk for a reasonable amount of
time after job-completion.
> "0" value seen in PigStat's map/reduce runtime, even when the job is
> successful
> -------------------------------------------------------------------------------
>
> Key: PIG-1829
> URL: https://issues.apache.org/jira/browse/PIG-1829
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.0
> Reporter: Thejas M Nair
> Fix For: 0.9.0
>
>
> Pig runtime calls JobClient.getMapTaskReports(jobId) and
> JobClient.getReduceTaskReports(jobId) to get statistics about numbers of
> maps/reducers, as well as max/min/avg time of these tasks. But from time to
> time, these calls return empty lists. When that happens pig is reports 0
> values for the stats.
> The jobtracker keeps the stats information only for a limited duration based
> on the configuration parameters mapred.jobtracker.completeuserjobs.maximum
> and mapred.job.tracker.retiredjobs.cache.size. Since pig collects the stats
> after jobs have finished running, it is possible that the stats for the
> initial jobs are no longer available. To have better chances of getting the
> stats, it should be collected as soon as the job is over.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
