Hi Christoph, I also enabled the detailed data, and the reports are very similar. As I said, this extra memory doesn't seem to be tracked by NMT. If you look at the summary, it says that it is only 1.1GB of committed memory, which mean the RES size should be below that, like when annotations are disabled.
Full GC doesn't help, again because this memory is off-heap. The summaries from the previous emails were generated immediately after restarting the Jetty (waited for the server to accept requests). This is a production environment, so we restart Jetty only when deploying bug fixes. Ciprian On Fri, Aug 28, 2015 at 11:56 AM, Christoph Läubrich <[email protected]> wrote: > Okay, have you tried enabling detailed report? > > "*Get detail data:* To get a more detailed view of native memory usage, > start the JVM with command line option: -XX:NativeMemoryTracking=detail. > This will track exactly what methods allocate the most memory. Enabling NMT > will result in 5-10 percent JVM performance drop and memory usage for NMT > adds 2 words to all malloc memory as malloc header. NMT memory usage is > also tracked by NMT." > > how long are you running the process and have you tried to force a Full-GC > (e.g. via JMX console). > > > Am 28.08.2015 10:50, schrieb Ciprian Hacman: > > Out Java team tried to debug this issue using Java tools. The problem we > encountered is that this extra memory is not tracked by Java at all (off > heap memory). We realised this using NMT - > https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html. > > > There was not change in NMT tracked memory with and without annotations > enabled. Please see the output of "jcmd <PID> VM.native_memory summary > scale=MB" below: > > Glassfish + annotations disabled (*551* MB RES on startup): > Total: reserved=2397MB, committed=*1159MB* > - Java Heap (reserved=1024MB, committed=1024MB) > (mmap: reserved=1024MB, committed=1024MB) > > - Class (reserved=1077MB, committed=60MB) > (classes #10470) > (malloc=1MB #15133) > (mmap: reserved=1076MB, committed=59MB) > > - Thread (reserved=23MB, committed=23MB) > (thread #80) > (stack: reserved=23MB, committed=23MB) > > - Code (reserved=249MB, committed=28MB) > (malloc=5MB #8188) > (mmap: reserved=244MB, committed=23MB) > > - GC (reserved=3MB, committed=3MB) > (mmap: reserved=3MB, committed=3MB) > > - Internal (reserved=3MB, committed=3MB) > (malloc=3MB #13793) > > - Symbol (reserved=14MB, committed=14MB) > (malloc=10MB #111702) > (arena=4MB #1) > > - Native Memory Tracking (reserved=2MB, committed=2MB) > (tracking overhead=2MB) > > Glassfish + annotations (*1909* MB RES on startup): > Total: reserved=2400MB, committed=*1162MB* > - Java Heap (reserved=1024MB, committed=1024MB) > (mmap: reserved=1024MB, committed=1024MB) > > - Class (reserved=1079MB, committed=61MB) > (classes #10552) > (malloc=1MB #16204) > (mmap: reserved=1078MB, committed=59MB) > > - Thread (reserved=22MB, committed=22MB) > (thread #74) > (stack: reserved=22MB, committed=22MB) > > - Code (reserved=249MB, committed=30MB) > (malloc=6MB #8349) > (mmap: reserved=244MB, committed=25MB) > > - GC (reserved=3MB, committed=3MB) > (mmap: reserved=3MB, committed=3MB) > > - Internal (reserved=3MB, committed=3MB) > (malloc=3MB #13773) > > - Symbol (reserved=14MB, committed=14MB) > (malloc=11MB #112818) > (arena=4MB #1) > > - Native Memory Tracking (reserved=3MB, committed=3MB) > (tracking overhead=2MB) > > - Arena Chunk (reserved=1MB, committed=1MB) > (malloc=1MB) > > Apache + annotations (*1900* MB RES on startup): > Total: reserved=2400MB, committed=*1163MB* > - Java Heap (reserved=1024MB, committed=1024MB) > (mmap: reserved=1024MB, committed=1024MB) > > - Class (reserved=1079MB, committed=61MB) > (classes #10679) > (malloc=1MB #16766) > (mmap: reserved=1078MB, committed=60MB) > > - Thread (reserved=23MB, committed=23MB) > (thread #80) > (stack: reserved=23MB, committed=23MB) > > - Code (reserved=249MB, committed=30MB) > (malloc=6MB #8492) > (mmap: reserved=244MB, committed=25MB) > > - GC (reserved=3MB, committed=3MB) > (mmap: reserved=3MB, committed=3MB) > > - Internal (reserved=3MB, committed=3MB) > (malloc=3MB #14043) > > - Symbol (reserved=14MB, committed=14MB) > (malloc=11MB #113587) > (arena=4MB #1) > > - Native Memory Tracking (reserved=3MB, committed=3MB) > (tracking overhead=2MB) > > Thanks, > Ciprian > > > > On Fri, Aug 28, 2015 at 11:20 AM, Christoph Läubrich < > [email protected]> wrote: > >> In general you should use a proper tool (Eclipse MAT or JProfiler for >> example) to investigate where how much memory is used. >> >> Depending on your application, annotation parsing might involve a lot of >> (extra) class-loading and information caching... but this is just a wild >> guess! >> >> >> Am 28.08.2015 10:03, schrieb Ciprian Hacman: >> >> Hi, >> >> Just wanted to make things a little more clear. >> >> We changed from Apache to Glassfish, but we disabled the "annotations" >> module to reduce the memory usage. We had to move to Glassfish because the >> Apache engine did not work without "annotations". >> >> To make things easier to follow, here are some numbers for our webapp >> (numbers represent RES memory): >> >> 1. apache + annotations - 2.5GB >> 2. apache + annotations disabled - not working >> 3. glassfish + annotations - 2.5GB >> 4. glassfish + annotations disabled - 750MB >> >> >> Not sure why annotations make this huge difference for us and how to >> debug further. >> I see some other people bumped into similar issues (which gave us this >> idea): >> >> https://www.eclipse.org/forums/index.php/t/1067502/ >> >> >> If anyone has any idea here, we would be happy to give it a try. >> >> Thanks, >> Ciprian >> >> >> >> _______________________________________________ >> jetty-users mailing list >> [email protected] >> To change your delivery options, retrieve your password, or unsubscribe >> from this list, visit >> https://dev.eclipse.org/mailman/listinfo/jetty-users >> > > > _______________________________________________ > jetty-users mailing [email protected] > To change your delivery options, retrieve your password, or unsubscribe from > this list, visithttps://dev.eclipse.org/mailman/listinfo/jetty-users > > > > _______________________________________________ > jetty-users mailing list > [email protected] > To change your delivery options, retrieve your password, or unsubscribe > from this list, visit > https://dev.eclipse.org/mailman/listinfo/jetty-users > -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/
_______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
