[
https://issues.apache.org/jira/browse/PIG-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565975#comment-13565975
]
Koji Noguchi commented on PIG-3148:
-----------------------------------
I cannot attach the original query, but to give you an idea
{noformat}
A = LOAD '$INPUT' USING MyLoader('\u0001') AS
( val1, val2, val3, val4, val5, val6, val7, val8, val9, val10, val11, val12,
val13, val14, val15, val16);
B = GROUP A BY (val1, val2, val3, val8) PARALLEL $NUM_REDUCERS;
C = FOREACH B {
D = FILTER A BY (val3 == 'status1' AND val5 == 'status2');
E = D.val4;
F = DISTINCT E;
G = FILTER D BY val7 == 'status3';
GENERATE group.val1, group.val2, group.val8, COUNT(F), COUNT(G), SUM(G.val9),
SUM(G.val10), SUM(G.val11), SUM(A.val12), SUM(A.val13), SUM(A.val15),
SUM(A.val14), SUM(A.val16);
}
STORE C INTO '$OUTPUT' USING PigStorage('\u0001');
{noformat}
Assuming that this script does not require a two huge DefaultDataBag, looked
into SpillableMemoryManager.handleNotification. 'handleNotification' is
called whenever certain memory condition is met but not necessary after the
gc().
What was happening on this user's case was,
(i) 400MB of DefaultDataBag#1 goes stale.
(ii) SpillableMemoryManager.handleNotification is triggered.
(iii) Since gc() is not called yet, WeakReference is still valid and pig
decides to spill holding the lock of this DefaultDataBag#1.mContents(ArrayList).
(iv) While reduce task is working on another 400MB DefaultDataBag#2, jvm heap
gets full and gc is called. Even though no one is using the stale
DefaultDataBag#1, it cannot be GC-ed since spill is holding the lock.
As a result, we end up with two DefaultDataBag leading to OOM.
> OutOfMemory exception while spilling stale DefaultDataBag. Extra option to
> gc() before spilling large bag.
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-3148
> URL: https://issues.apache.org/jira/browse/PIG-3148
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
>
> Our user reported that one of their jobs in pig 0.10 occasionally failed with
> 'Error: GC overhead limit exceeded' or 'Error: Java heap space', but
> rerunning it sometimes finishes successfully.
> For 1G heap reducer, heap dump showed it contained two huge DefaultDataBag
> with 300-400MBytes each when failing with OOM.
> Jstack at the time of OOM always showed that spill was running.
> {noformat}
> "Low Memory Detector" daemon prio=10 tid=0xb9c11800 nid=0xa52 runnable
> [0xb9afc000]
> java.lang.Thread.State: RUNNABLE
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> - locked <0xe57c6390> (a java.io.BufferedOutputStream)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> - locked <0xe57c60b8> (a java.io.DataOutputStream)
> at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
> at org.apache.pig.data.utils.SedesHelper.writeBytes(SedesHelper.java:46)
> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:537)
> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
> at
> org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
> at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
> at org.apache.pig.data.DefaultDataBag.spill(DefaultDataBag.java:106)
> - locked <0xceb16190> (a java.util.ArrayList)
> at
> org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:243)
> - locked <0xbeb86318> (a java.util.LinkedList)
> at
> sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:138)
> at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
> at
> sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
> at sun.management.Sensor.trigger(Sensor.java:120)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
