[
https://issues.apache.org/jira/browse/GROOVY-7498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629469#comment-14629469
]
Max Kuchin commented on GROOVY-7498:
------------------------------------
I think it some internal JVM structures. What kind of tool can I use to track
JVM native memory other than jcmd VM.native_memory?
I'd made deeper investigation with JVM NativeMemoryTracking enabled. I'd made a
baseline 1 day after start of tomcat with grails web app with command "jcmd pid
VM.native_memory baseline" and get detailed diff on 6th day with command "jcmd
pid VM.native_memory detail.diff".
Here is results of command:
Native Memory Tracking:
Total: reserved=7766583KB +380523KB, committed=7767515KB +382611KB
- Java Heap (reserved=6602752KB, committed=6602752KB)
(mmap: reserved=6602752KB, committed=6602752KB)
- Class (reserved=7857KB +1425KB, committed=7857KB +1425KB)
(classes #24501 +1011)
(malloc=7857KB +1425KB, #91299 +5028)
- Thread (reserved=553KB +5KB, committed=553KB +5KB)
(thread #109 -222)
(stack: reserved=112052KB +1028KB,
committed=113080KB +1028KB)
(malloc=344KB +3KB, #443 +4)
(arena=209KB +2KB, #218 +2)
- Code (reserved=60008KB +570KB, committed=59944KB +602KB)
(malloc=10088KB +570KB, #12570 +496)
(mmap: reserved=49920KB, committed=49856KB +32KB)
- GC (reserved=546647KB +91360KB, committed=546647KB
+93416KB)
(malloc=301603KB +91360KB, #227847 +114771)
(mmap: reserved=245044KB, committed=245044KB
+2056KB)
- Compiler (reserved=611KB +15KB, committed=611KB +15KB)
(malloc=513KB +15KB, #2621 +15)
(arena=98KB, #2)
- Internal (reserved=346612KB +248665KB, committed=346580KB
+248665KB)
(malloc=346580KB +248665KB, #1001951 +767996)
(mmap: reserved=32KB, committed=0KB)
- Symbol (reserved=22463KB +133KB, committed=22463KB +133KB)
(malloc=18268KB +133KB, #187153 +1271)
(arena=4195KB, #1)
- Memory Tracking (reserved=65046KB +35529KB, committed=65046KB
+35529KB)
(malloc=65046KB +35529KB, #444 +64)
- Pooled Free Chunks (reserved=1982KB +1793KB, committed=1982KB +1793KB)
(malloc=1982KB +1793KB)
Details:
[0x00007f8119c8dfa8] CHeapObj<(unsigned short)1280>::operator new(unsigned
long, unsigned char*)+0x28
(malloc=20208KB +9906KB, #216639 +112184)
[0x00007f8119c960c7] ResourceObj::operator new(unsigned long,
ResourceObj::allocation_type, unsigned short)+0x87
(malloc=99KB +16KB, #3147 +514)
[0x00007f8119c970b3] ChunkPool::allocate(unsigned long,
AllocFailStrategy::AllocFailEnum)+0x93
(malloc=6124KB +1795KB, #412 +60)
[0x00007f8119c97128] CHeapObj<(unsigned short)1792>::operator new(unsigned
long, unsigned char*)+0x28
(malloc=231487KB +192917KB, #782059 +649847)
[0x00007f8119ca320c] AllocateHeap(unsigned long, unsigned short, unsigned
char*, AllocFailStrategy::AllocFailEnum)+0x2c
(malloc=148833KB +407KB, #9798 +6855)
[0x00007f8119d06ce9] ArrayAllocator<unsigned long, (unsigned
short)1792>::allocate(unsigned long)+0x49
(malloc=111381KB +55262KB, #207368 +110644)
[0x00007f8119e00fc8] CHeapObj<(unsigned short)256>::operator new(unsigned long,
unsigned char*)+0x28
(malloc=1886KB +173KB, #86636 +4018)
[0x00007f8119e141d9] CodeBlob::set_oop_maps(OopMapSet*)+0x49
(malloc=9895KB +536KB, #11010 +372)
[0x00007f8119e1b648] CHeapObj<(unsigned short)1024>::operator new(unsigned
long, unsigned char*)+0x28
(malloc=124KB +34KB, #1321 +124)
[0x00007f8119e99552] Deoptimization::fetch_unroll_info_helper(JavaThread*)+0x9b2
(malloc=166KB +8KB, #60 +3)
[0x00007f8119f8190d] GenericGrowableArray::raw_allocate(int)+0x9d
(malloc=476KB +95KB, #3137 +514)
[0x00007f811a140de7]
LoaderConstraintTable::extend_loader_constraint(LoaderConstraintEntry*, Handle,
klassOopDesc*)+0xb7
(malloc=111KB +15KB, #298 +31)
[0x00007f811a190918] MemRecorder::MemRecorder()+0x208
(malloc=5096KB +769KB, #424 +64)
[0x00007f811a197f54] MemPointerArrayImpl<MemPointerRecordEx>::is_full()+0x44
(malloc=59600KB +34760KB, #1)
[0x00007f811a20ab95] OopMapCache::OopMapCache()+0x45
(malloc=4404KB +1186KB, #3523 +949)
[0x00007f811a236d8b] ParkEvent::operator new(unsigned long)+0x3b
(malloc=370KB +78KB, #776 +164)
[0x00007f811a28394f] PtrQueueSet::allocate_buffer()+0xff
(malloc=9893KB +6194KB, #1863 +1049)
[0x00007f811a2cd115] RSHashTable::RSHashTable(unsigned long)+0x85
(malloc=119735KB +68403KB, #3075 +512)
[0x00007f811a2cd15e] RSHashTable::RSHashTable(unsigned long)+0xce
(malloc=11974KB +6840KB, #3075 +512)
[0x00007f811a30f0f3] Symbol::operator new(unsigned long, int, Thread*)+0x53
(malloc=9927KB +69KB, #186181 +1263)
[0x00007f811a34e756] Thread::allocate(unsigned long, bool, unsigned short)+0x56
(malloc=310KB +3KB, #108 +1)
[0x00007f811a371135] Unsafe_AllocateMemory+0x115
(malloc=140KB +20KB, #31 +10)
[0x00007f811a38e7f1] vframeArray::allocate(JavaThread*, int,
GrowableArray<compiledVFrame*>*, RegisterMap*, frame, frame, frame)+0x101
(malloc=121KB +6KB, #60 +3)
[0x00007f811a34fdd4] JavaThread::run()+0x24
(mmap: reserved=96632KB +1028KB, committed=97660KB
+1028KB)
[0x00007f811a391eb6] ReservedSpace::initialize(unsigned long, unsigned long,
bool, c+0x266
(mmap: reserved=14272KB, committed=14208KB +2088KB)
Biggest rises in memory consumption was "CHeapObj<(unsigned
short)1280>::operator new" and "CHeapObj<(unsigned short)1792>::operator new"
structures.
Internal memory consumption with groovy-all-2.3.7.jar after first day is almost
flat. I can also reproduce same behaviour with groovy-all-2.3.8.jar.
> Native memory leak
> ------------------
>
> Key: GROOVY-7498
> URL: https://issues.apache.org/jira/browse/GROOVY-7498
> Project: Groovy
> Issue Type: Bug
> Affects Versions: 2.3.11
> Environment: Ubuntu 14.04 , grails 2.4.5 java 1.7.0_80, tomcat 7.0.56
> Reporter: Max Kuchin
>
> We are experiencing JVM native memory leak after upgrading our web
> application from grails 2.4.4 (groovy 2.3.7) to grails 2.4.5 (groovy 2.3.11).
> It disappears if we replace groovy-all-2.3.11.jar in web application with
> groovy-all-2.3.7.jar.
> As a result of the leak JVM native memory usage can grow to 4G in two weeks
> after start of web application.
> We'd enabled native memory tracking (-XX:NativeMemoryTracking=detail) to
> gather some details. Here are results of native memory summary snapshot made
> with command jcmd pid VM.native_memory summary
> Native memory summary after start of application:
> Total: reserved=6039782KB, committed=6039342KB
> - Java Heap (reserved=5447680KB, committed=5447680KB)
> (mmap: reserved=5447680KB, committed=5447680KB)
> - Class (reserved=2944KB, committed=2944KB)
> (classes #19490)
> (malloc=2944KB, #56242)
> - Thread (reserved=131202KB, committed=131202KB)
> (thread #127)
> (stack: reserved=130556KB, committed=130556KB)
> (malloc=402KB, #515)
> (arena=244KB, #254)
> - Code (reserved=54669KB, committed=54261KB)
> (malloc=4749KB, #7615)
> (mmap: reserved=49920KB, committed=49512KB)
> - GC (reserved=350261KB, committed=350261KB)
> (malloc=148081KB, #28742)
> (mmap: reserved=202180KB, committed=202180KB)
> - Compiler (reserved=399KB, committed=399KB)
> (malloc=301KB, #480)
> (arena=98KB, #2)
> - Internal (reserved=14366KB, committed=14334KB)
> (malloc=14334KB, #21616)
> (mmap: reserved=32KB, committed=0KB)
> - Symbol (reserved=20911KB, committed=20911KB)
> (malloc=17355KB, #180610)
> (arena=3556KB, #1)
> - Memory Tracking (reserved=14986KB, committed=14986KB)
> (malloc=14986KB, #492)
> - Pooled Free Chunks (reserved=2365KB, committed=2365KB)
> (malloc=2365KB)
> One day after start:
>
> Total: reserved=6610742KB , committed=6610518KB
> - Java Heap (reserved=5447680KB, committed=5447680KB)
> (mmap: reserved=5447680KB, committed=5447680KB)
> - Class (reserved=5525KB, committed=5525KB)
> (classes #21209)
> (malloc=5525KB, #71960)
> - Thread (reserved=646KB, committed=646KB)
> (thread #127)
> (stack: reserved=130556KB, committed=130556KB)
> (malloc=402KB +13KB, #515)
> (arena=244KB +8KB, #254)
> - Code (reserved=58182KB, committed=57990KB)
> (malloc=8262KB, #11074)
> (mmap: reserved=49920KB, committed=49728KB)
> - GC (reserved=423906KB, committed=423906KB)
> (malloc=221726KB, #82681)
> (mmap: reserved=202180KB, committed=202180KB)
> - Compiler (reserved=440KB, committed=440KB)
> (malloc=342KB +20KB, #513)
> (arena=98KB, #2)
> - Internal (reserved=472055KB, committed=472023KB)
> (malloc=472023KB, #746397)
> (mmap: reserved=32KB, committed=0KB)
> - Symbol (reserved=21668KB +7KB, committed=21668KB +7KB)
> (malloc=17760KB +7KB, #185311)
> (arena=3908KB, #1)
> - Memory Tracking (reserved=49894KB, committed=49894KB)
> (malloc=49894KB +2464KB, #536)
> - Pooled Free Chunks (reserved=191KB, committed=191KB)
> (malloc=191KB)
> There is substantial increase in Internal memory usage and object count:
> 14366KB, #21616 -> 472023KB, #746397
> Native memory summary, one day after start, same application, different
> groovy version (groovy-all-2.3.7.jar):
> Total: reserved=6098509KB, committed=6099309KB
> - Java Heap (reserved=5447680KB, committed=5447680KB)
> (mmap: reserved=5447680KB, committed=5447680KB)
> - Class (reserved=6263KB, committed=6263KB)
> (classes #21179)
> (malloc=6263KB, #71507)
> - Thread (reserved=117771KB, committed=118799KB)
> (thread #114)
> (stack: reserved=117192KB, committed=118220KB)
> (malloc=360KB, #463)
> (arena=219KB, #228)
> - Code (reserved=58131KB, committed=57935KB)
> (malloc=8211KB, #10858)
> (mmap: reserved=49920KB, committed=49724KB)
> - GC (reserved=383041KB, committed=383041KB)
> (malloc=180861KB, #80128)
> (mmap: reserved=202180KB, committed=202180KB)
> - Compiler (reserved=268KB, committed=268KB)
> (malloc=170KB, #313)
> (arena=98KB, #2)
> - Internal (reserved=41336KB, committed=41304KB)
> (malloc=41304KB, #69639)
> (mmap: reserved=32KB, committed=0KB)
> - Symbol (reserved=21696KB, committed=21696KB)
> (malloc=17788KB, #185505)
> (arena=3908KB, #1)
> - Memory Tracking (reserved=22135KB, committed=22135KB)
> (malloc=22135KB, #432)
> - Pooled Free Chunks (reserved=189KB, committed=189KB)
> (malloc=189KB)
> Internal memory consumption, one day after start, comparing groovy 2.3.7 to
> groovy 2.3.11:
> 41336KB, #69639 - groovy 2.3.7
> 472023KB, #746397 - groovy 2.3.11, more than 10 times increase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
