[
https://issues.apache.org/jira/browse/PIG-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13787954#comment-13787954
]
Vivek Padmanabhan commented on PIG-3486:
----------------------------------------
Hi Ankit/Daniel,
Thanks for looking into this.
Meanwhile, is the current patch safe to be applied to our cluster for Pig
0.11.1 . All basic tests are not having any issues, not sure of any cases that
I have missed.
> Pig hitting OOM while using PigRunner.run()
> -------------------------------------------
>
> Key: PIG-3486
> URL: https://issues.apache.org/jira/browse/PIG-3486
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.11.1
> Reporter: Vivek Padmanabhan
> Attachments: histolive.txt, PIG-3486.patch1
>
>
> I have a timer based class, which will trigger a pig script execution every 5
> minutes using PigRunner.run(args, null).
> But it looks like the heap usage is gradually increasing after around 15days
> it crossed 1G, ie after invoking the above method 4k times.
> The top entries of the histo live goes like this;
> num #instances #bytes class name
> ----------------------------------------------
> 1: 2430178 433053080 [C
> 2: 3055280 97768960 java.util.Hashtable$Entry
> 3: 2454870 78555840 java.lang.String
> 4: 1585204 50726528 java.util.HashMap$Entry
> 5: 260310 37503984 <constMethodKlass>
> 6: 260310 35413536 <methodKlass>
> 7: 35024 23724672 [Ljava.util.Hashtable$Entry;
> 8: 7599 18141016 <constantPoolKlass>
> 9: 47551 18066696 [Ljava.util.HashMap$Entry;
> 10: 209516 16761280 java.lang.reflect.Method
> 11: 212292 16732008 [I
> 12: 6881 11332896 <constantPoolCacheKlass>
> 13: 7599 7160920 <instanceKlassKlass>
> 14: 79412 4447072 java.util.ResourceBundle$CacheKey
> 15: 10787 3958464 [S
> 16: 79412 3811776 java.util.ResourceBundle$BundleReference
> 17: 26634 3458160 [B
> 18: 133701 3208824 java.util.LinkedList$Node
> 19: 85492 2735744
> java.util.concurrent.ConcurrentHashMap$HashEntry
> 20: 79412 2541184 java.util.ResourceBundle$LoaderReference
> 21: 47515 2280720 java.util.HashMap
> 22: 37298 2274416 [Ljava.lang.Object;
> 23: 70638 2260416 java.util.LinkedList
> 24: 2949 1994376 <methodDataKlass>
> 25: 7914 1749080 java.lang.Class
> 26: 62746 1505904
> org.apache.commons.logging.impl.Log4JLogger
> 27: 16639 1463824 [[I
> 28: 21279 1361856 java.net.URL
> 29: 28090 1348320 java.util.Hashtable
> 30: 14167 1231856 [Ljava.util.WeakHashMap$Entry;
> 31: 17770 710800 java.lang.ref.Finalizer
> 32: 10626 680064 java.util.jar.JarFile
> 33: 14167 680016 java.util.WeakHashMap
> 34: 14238 569520 java.util.WeakHashMap$Entry
> 35: 7104 568320 java.util.jar.JarFile$JarFileEntry
> 36: 165 567264
> [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
> 37: 10637 510576 sun.nio.cs.UTF_8$Encoder
> 38: 10633 510384 sun.misc.URLClassPath$JarLoader
> 39: 14176 453632 java.lang.ref.ReferenceQueue
> 40: 17747 409752 [Ljava.lang.Class;
> 41: 3463 387856
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$HangingJobKiller
> 42: 15355 368520 java.util.ArrayList
> 43: 10632 340224 java.util.zip.ZipCoder
> 44: 6932 332736 java.util.Properties
> 45: 4060 292320 java.lang.reflect.Constructor
> 46: 7143 285720 java.util.LinkedHashMap$Entry
> 47: 3517 281360
> org.apache.pig.impl.PigContext$ContextClassLoader
> 48: 3476 278144
> [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;
> 49: 3458 276640 java.net.URI
> 50: 8576 274432 antlr.ANTLRHashString
> 51: 10632 255168 java.util.ArrayDeque
> There are way too many instances of MapReduceLauncher$HangingJobKiller.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
