[
https://issues.apache.org/jira/browse/PIG-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-3148:
------------------------------
Attachment: pig-3148-v01.patch
Uploading a patch that adds a feature that would call System.gc() when
Spillable is bigger than 'pig.spill.extragc.size.threshold' before spilling.
This extra gc() is called at most once per handleNotification and also disabled
as default since adding GC has a risk of changing the performance drastically.
For the job I was looking, adding
'-Dpig.spill.extragc.size.threshold=100000000' let the job run successfully
with no OOM errors.
(Note: Separate spill issues before this patch on 0.11 is tracked at PIG-3147.)
> OutOfMemory exception while spilling stale DefaultDataBag. Extra option to
> gc() before spilling large bag.
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-3148
> URL: https://issues.apache.org/jira/browse/PIG-3148
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Attachments: pig-3148-v01.patch
>
>
> Our user reported that one of their jobs in pig 0.10 occasionally failed with
> 'Error: GC overhead limit exceeded' or 'Error: Java heap space', but
> rerunning it sometimes finishes successfully.
> For 1G heap reducer, heap dump showed it contained two huge DefaultDataBag
> with 300-400MBytes each when failing with OOM.
> Jstack at the time of OOM always showed that spill was running.
> {noformat}
> "Low Memory Detector" daemon prio=10 tid=0xb9c11800 nid=0xa52 runnable
> [0xb9afc000]
> java.lang.Thread.State: RUNNABLE
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> - locked <0xe57c6390> (a java.io.BufferedOutputStream)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> - locked <0xe57c60b8> (a java.io.DataOutputStream)
> at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
> at org.apache.pig.data.utils.SedesHelper.writeBytes(SedesHelper.java:46)
> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:537)
> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
> at
> org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
> at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
> at org.apache.pig.data.DefaultDataBag.spill(DefaultDataBag.java:106)
> - locked <0xceb16190> (a java.util.ArrayList)
> at
> org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:243)
> - locked <0xbeb86318> (a java.util.LinkedList)
> at
> sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:138)
> at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
> at
> sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
> at sun.management.Sensor.trigger(Sensor.java:120)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
