Hello. I am trying to access custom counters that I have created in an mapreduce job on Yarn.
After job.waitForCompletion(true) call, I try to do job.getCounters() but I get a null. This only happens if I run a heavy job meaning a) a lot of data and b) lot of reducers. E.g. for 10million records with 20 reducers on a 10 node cluster it works. But on 60million records with 70 reducers on a 10 node cluster it doesn't. The job itself competes successfully. I did see the following related JIRAs. But the first one is for old version, pre-Yarn and those properties I think are not valid anymore. The second one does not seem to provide a solution? I tried using the suggested trick on the client side but no success there either. MAPREDUCE-1920 <https://issues.apache.org/jira/browse/MAPREDUCE-1920> MAPREDUCE-4442 <https://issues.apache.org/jira/browse/MAPREDUCE-4442> Please advise, how can I retrieve my custom counters after jobs completion. Am I missing something? Do I need to configure some job history stuff? I do a mention of ATS as well but I don't know how much that is applicable here. Thanks a lot. My version is: 2.3.0-cdh5.1.0 Regards, Shahab
