[
https://issues.apache.org/jira/browse/PIG-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated PIG-2313:
----------------------------
Attachment: PIG-2313-2.patch
Hi, Daniels,
Your fix for Reporter is right. We didn't pass reporter to reducer in
illustrate. I attached PIG-2313-2.patch also fix the "store func setup" issue.
I include your test case in patch.
> NPE in ILLUSTRATE trying to get StatusReporter in STORE
> -------------------------------------------------------
>
> Key: PIG-2313
> URL: https://issues.apache.org/jira/browse/PIG-2313
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.10
> Reporter: Doug Daniels
> Attachments: PIG-2313-2.patch, PIG-2313-example.tar.gz,
> PIG-2313-first_shot.patch
>
>
> I'm seeing an NPE trying to do an illustrate on a script. So far the
> simplest version of the script that exhibits the issue is:
> {code}
> raw = LOAD 'data.txt' USING PigStorage() AS (x:int, y:int);
> filtered = FILTER raw BY x < 5;
> grouped = GROUP filtered BY x;
> counted = FOREACH grouped
> GENERATE group AS x,
> COUNT(filtered) AS the_count;
> rmf output;
> STORE counted INTO 'output';
> {code}
> I had to pass a few nested Exceptions along to get it, but the bottom stack
> trace looks like:
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88)
> at
> org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:60)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createRecordCounter(MapReducePOStoreImpl.java:121)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:108)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.cleanup(PigGenericMapReduce.java:525)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
> at
> org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:222)
> at
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> at
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:275)
> at
> org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:418)
> ... 21 more
> {code}
> It looks like the IllustratorContext in the hadoop20 PigMapReduce.java shim
> (line 73) is getting setup with a null StatusReporter. This seems to be for
> the Reduce phase. On the other hand, the PigMapBase.java sets up the
> IllustratorContext with an IllustratorDummyReporter for the Map phase.
> Eventually when the code in MapReducePOStoreImpl line 121 tries to get the
> reporter, it fails with the NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
