[
https://issues.apache.org/jira/browse/PIG-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-2963:
-------------------------------
Attachment: PIG-2963.patch
This is a regression from PIG-2923. Verified that the error goes away by
reverting it.
What's happening is that illustrate adds to DefautAbstractBag a tuple that has
a ReadOnceBag field.
The problem is that PIG-2923 made DefautAbstractBag check whether or not it
should spill to disk every time a new element is added to it. To compute the
memory size of the new element, DefautAbstractBag iterates through every field
of the tuple, in this case which is ReadOnceBag. Unfortunately, ReadOnceBag
doesn't support getMemorySize() and throws a runtime exception.
I am attaching a simple fix that makes ReadOnceBag.getMemorySize() return 0
instead of throwing a runtime exception. I am returning 0 here because the
comment in ReadOnceBag says "this bag does not store the tuples in memory".
I am not familiar with ReadOnceBag, so please correct me if this is not a
proper fix.
Thanks!
> Illustrate command and POPackageLite
> ------------------------------------
>
> Key: PIG-2963
> URL: https://issues.apache.org/jira/browse/PIG-2963
> Project: Pig
> Issue Type: Bug
> Reporter: Allan Avendaño
> Priority: Critical
> Fix For: 0.11
>
> Attachments: PIG-2963.patch
>
>
> While trying to execute a simple script like:
> A = LOAD 'test01' AS (f1:chararray,f2:int,f3:chararray);
> B = order A by f1;
> illustrate B;
> or
> C = foreach B generate f1, f2;
> illustrate C;
> I got the following exception:
> java.lang.RuntimeException: ReadOnceBag does not support getMemorySize
> operation
> at org.apache.pig.data.ReadOnceBag.getMemorySize(ReadOnceBag.java:74)
> at org.apache.pig.data.SizeUtil.getPigObjMemSize(SizeUtil.java:61)
> at org.apache.pig.data.DefaultTuple.getMemorySize(DefaultTuple.java:180)
> at
> org.apache.pig.pen.util.ExampleTuple.getMemorySize(ExampleTuple.java:97)
> at
> org.apache.pig.data.DefaultAbstractBag.getMemorySize(DefaultAbstractBag.java:148)
> at
> org.apache.pig.data.DefaultAbstractBag.markSpillableIfNecessary(DefaultAbstractBag.java:100)
> at
> org.apache.pig.data.DefaultAbstractBag.add(DefaultAbstractBag.java:92)
> at org.apache.pig.pen.Illustrator.addData(Illustrator.java:116)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.illustratorMarkup(POPackageLite.java:227)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.getNext(POPackageLite.java:182)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:422)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> at
> org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:235)
> at
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> at
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
> at
> org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
> at
> org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:98)
> at
> org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
> at org.apache.pig.PigServer.getExamples(PigServer.java:1180)
> at
> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:738)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:626)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:323)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> at org.apache.pig.Main.run(Main.java:538)
> at org.apache.pig.Main.main(Main.java:154)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> ======================================================================
> At log file, the following:
> Pig Stack Trace
> ---------------
> ERROR 2997: Encountered IOException. Exception
> java.io.IOException: Exception
> at org.apache.pig.PigServer.getExamples(PigServer.java:1186)
> at
> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:738)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:626)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:323)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> at org.apache.pig.Main.run(Main.java:538)
> at org.apache.pig.Main.main(Main.java:154)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.RuntimeException: ReadOnceBag does not support
> getMemorySize operation
> at org.apache.pig.data.ReadOnceBag.getMemorySize(ReadOnceBag.java:74)
> at org.apache.pig.data.SizeUtil.getPigObjMemSize(SizeUtil.java:61)
> at
> org.apache.pig.data.DefaultTuple.getMemorySize(DefaultTuple.java:180)
> at
> org.apache.pig.pen.util.ExampleTuple.getMemorySize(ExampleTuple.java:97)
> at
> org.apache.pig.data.DefaultAbstractBag.getMemorySize(DefaultAbstractBag.java:148)
> at
> org.apache.pig.data.DefaultAbstractBag.markSpillableIfNecessary(DefaultAbstractBag.java:100)
> at
> org.apache.pig.data.DefaultAbstractBag.add(DefaultAbstractBag.java:92)
> at org.apache.pig.pen.Illustrator.addData(Illustrator.java:116)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.illustratorMarkup(POPackageLite.java:227)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackageLite.getNext(POPackageLite.java:182)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:422)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> at
> org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:235)
> at
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> at
> org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
> at
> org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
> at
> org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:98)
> at
> org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
> at org.apache.pig.PigServer.getExamples(PigServer.java:1180)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
