[
https://issues.apache.org/jira/browse/PIG-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587246#comment-13587246
]
Jarek Jarcec Cecho commented on PIG-3002:
-----------------------------------------
Hi [~billgraham],
thank you very much for taking a look on this Jira and my patch. I was
considering similar solution as you proposed in my early work, but I've notice
one side effect during experiments with my early patches.
I've created quite pathological case when my cluster was using default
configuration, but I've limit the number of allowed counters to 3 on machine
where I've executed pig. I've noticed that with similar fix, pig will print out
couple of counters and than bail out on exception on first non existing
Counter. As a result not all the counters will be printed out even though they
are available in the {{Couter}} object.
My experiment is obviously not entirely real as it's unlikely that users will
have different hadoop configuration. However I believe that it model the edge
situation when mapreduce job will create almost all available counters, but
because client is iterating over predefined set, not all of them will be
printed out.
I've also did one step further and put the {{try-catch}} block inside the
{{for}} iteration. I've noticed that in this situation we might print out the
error message several times, which is kind of distracting. This lead me to the
idea of doing the changes on the shim layer that I've submitted.
Jarcec
> Pig client should handle CountersExceededException
> --------------------------------------------------
>
> Key: PIG-3002
> URL: https://issues.apache.org/jira/browse/PIG-3002
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Jarek Jarcec Cecho
> Labels: newbie, simple
> Attachments: PIG-3002.2.patch, PIG-3002.patch
>
>
> Running a pig job that uses more than 120 counters will succeed, but a grunt
> exception will occur when trying to output counter info to the console. This
> exception should be caught and handled with friendly messaging:
> {noformat}
> org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected
> error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1275)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> at org.apache.pig.PigServer.execute(PigServer.java:1239)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:333)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:136)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:197)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:604)
> at org.apache.pig.Main.main(Main.java:154)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: org.apache.hadoop.mapred.Counters$CountersExceededException:
> Error: Exceeded limits on number of counters - Counters=120 Limit=120
> at
> org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:312)
> at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:431)
> at org.apache.hadoop.mapred.Counters.getCounter(Counters.java:495)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.computeWarningAggregate(MapReduceLauncher.java:707)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:442)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
