[
https://issues.apache.org/jira/browse/PIG-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13549013#comment-13549013
]
Jarek Jarcec Cecho commented on PIG-3002:
-----------------------------------------
Hi Bill,
thank you very much for your review, I appreciate your time. My first
implementation actually took the way you've proposed. But after that I dig into
the logic and realized that it might not be necessary. As far as I understand
purpose of the method computeWarningAggregate() is to aggregate existing
counters from predefined Enum PigWarning. Therefore this method should not
create any new counters. However the method call Counters.getCounter(Enum) will
create new counter in case that it's not there (as side effect) and throw
CountersExceededException in case that it can't create it.
That's why I've introduced getCounterValue() method on the shim layer that
silently ignore this exception. The purpose of this method is only to get the
value, not create the counter. And if we got the exception, than it means that
the counter is not there (and can't be created) and thus it seems to me that
returning 0 should be valid operation at this point.
I believe that overall aggregates should be valid.
Does this reasoning make any sense?
Jarcec
> Pig client should handle CountersExceededException
> --------------------------------------------------
>
> Key: PIG-3002
> URL: https://issues.apache.org/jira/browse/PIG-3002
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Labels: newbie, simple
> Attachments: PIG-3002.patch
>
>
> Running a pig job that uses more than 120 counters will succeed, but a grunt
> exception will occur when trying to output counter info to the console. This
> exception should be caught and handled with friendly messaging:
> {noformat}
> org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected
> error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1275)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> at org.apache.pig.PigServer.execute(PigServer.java:1239)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:333)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:136)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:197)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:604)
> at org.apache.pig.Main.main(Main.java:154)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: org.apache.hadoop.mapred.Counters$CountersExceededException:
> Error: Exceeded limits on number of counters - Counters=120 Limit=120
> at
> org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:312)
> at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:431)
> at org.apache.hadoop.mapred.Counters.getCounter(Counters.java:495)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.computeWarningAggregate(MapReduceLauncher.java:707)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:442)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
