I am running a hadoop job written in PIG. It fails from out of memory because a 
UDF function consumes a lot of memory, it loads a big file. What are the 
settings to avoid the following OutOfMemoryError? I guess by simply giving PIG 
big memory (java -XmxBIGmemory org.apache.pig.Main ...) won't work.

Error message --->

java.lang.OutOfMemoryError: Java heap space
        at java.util.regex.Pattern.compile(Pattern.java:1451)
        at java.util.regex.Pattern.(Pattern.java:1133)
        at java.util.regex.Pattern.compile(Pattern.java:823)
        at java.lang.String.split(String.java:2293)
        at java.lang.String.split(String.java:2335)
        at UDF.load(Unknown Source)
        at UDF.load(Unknown Source)
        at UDF.exec(Unknown Source)
        at UDF.exec(Unknown Source)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:201)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:287)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:278)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:204)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:231)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:240)
        at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:249)
        at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:240)
        at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:93)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
        at org.apache.hadoop.mapred.Child.main(Child.java:155)

Thanks!
Michael



      

Reply via email to