Illustrate throws NullPointerException when custom load function uses counters
------------------------------------------------------------------------------
Key: PIG-2481
URL: https://issues.apache.org/jira/browse/PIG-2481
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.9.1
Environment: Apache Pig 0.9.1
JSON Loader from ElephantBird 2.1.1 (i.e. all changes through the following
commit:
https://github.com/kevinweil/elephant-bird/commit/13dda2a1de31844fe07bcf4a58e94fffcb9aa374)
Reporter: Matt Martin
Priority: Minor
Pig throws the following NullPointerException while trying to use "illustrate"
in conjunction with ElephantBird's JSON loader:
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:84)
at
org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:55)
at
com.twitter.elephantbird.pig.util.PigCounterHelper.incrCounter(PigCounterHelper.java:54)
at
com.twitter.elephantbird.pig.load.JsonLoader.incrCounter(JsonLoader.java:91)
at
com.twitter.elephantbird.pig.load.JsonLoader.getNext(JsonLoader.java:55)
at
org.apache.pig.impl.io.ReadToEndLoader.getNextHelper(ReadToEndLoader.java:210)
at
org.apache.pig.impl.io.ReadToEndLoader.getNext(ReadToEndLoader.java:190)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad.getNext(POLoad.java:129)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:267)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:262)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at
org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:194)
at
org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
at
org.apache.pig.pen.ExampleGenerator.readBaseData(ExampleGenerator.java:222)
at
org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:154)
at org.apache.pig.PigServer.getExamples(PigServer.java:1245)
at
org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:553)
at org.apache.pig.Main.main(Main.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
I have verified that this issue does not occur if the ElephantBird JSON loader
is not used so the issue seems to be related to the interaction between Pig and
a Custom Loader (in this case, it seems to be related to the way in which Pig's
"illustrate" functionality handles custom counters). I've also verified that
this issue does not occur with Pig 0.8.1 regardless of whether the ElephantBird
JSON loader is used or not, so this seems to be a Pig-specific issue (as
opposed to ElephantBird-related).
I'll attach sample input and script for reference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
