Hi,
I have a relatively large pig scripts (around 1.5k lines, 85 assignments).
Around 150 columns are getting projected, joined, grouped and aggregated
ending in multiple stores.
Pig 0.11.1 fails with the following error even before any jobs are fired:
Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error. Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirstLP(LogicalPlanPrinter.java:83)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.visit(LogicalPlanPrinter.java:69)
at
org.apache.pig.newplan.logical.relational.LogicalPlan.getSignature(LogicalPlan.java:122)
at org.apache.pig.PigServer.execute(PigServer.java:1237)
at org.apache.pig.PigServer.executeBatch(PigServer.java:333)
at
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:604)
at org.apache.pig.Main.main(Main.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Increasing heap size to 2Gb doesn't help either. The only thing that
appears to get the script working is to disable multi query optimization.
Has anyone else faced a similar problem with Pig running out of memory
while compiling the script? Any other way to get it to work besides
disabling multi-query optimization?
Thanks,
Shubham.
