I would start with moving this tomcat to its own hw. Did you check swap? This long pauses could be because part of your heap is swapped to hdd
Regards, Zdenek Henek On Tue, Nov 15, 2016, 21:37 George I. Develekos <gdevele...@omilia.com> wrote: > > On 15-Nov-16 10:22 PM, Christopher Schultz wrote: > > > > George, > > > > On 11/15/16 10:46 AM, George I. Develekos wrote: > >> Hello guys, > >> > >> We are having problems on a production system with very long "full > >> GC" times, as long as1200sec real time (!!!). > >> > >> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat > >> 7.0.64. > >> > >> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC > >> -XX:+CMSIncrementalMode > >> > >> No other custom memory-related settings are in place. > >> > >> Looking at the GC log, the last few Full-GC entries are: > >> > >> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K), > >> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm : > >> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times: > >> user=1.96 sys=0.13, real=2.11 secs] > >> > >> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark: > >> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01, > >> real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969: > >> [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times: > >> user=6.12 sys=1.36, real=*23.21 secs*] > >> > >> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K), > >> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm : > >> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times: > >> user=2.18 sys=0.14, real=2.34 secs] > >> > >> > >> What can we do? > > 1367.020 Full GC duration=2.11s > > 2579.317 Full GC duration=1212.49s > > > > So your full GC immediately started another full GC that took 20 minutes > > ? > > > > Are you only showing certain FULL GC activity from your log, or is > > that everything? > > > > CMS should have a mark and then a sweep each time, but your times > > don't seem to add up. > > > > also note that the whole point of CMS is that there isn't any > > stop-the-world during the mark portion of the process. > > > > Are you actually experiencing a problem, or are you just suffering > > from instrumentor's remorse? > > > > - -chris > > > > Chris, > > What I listed is the result of the command: > > grep "Full GC" gc.log > > So (obviously) I have skipped other GC activity, i.e. whatever GC > activity didn't include the "Full GC" string. > > Yes we are having app trouble due to the GC delays so this is a real > problem. Our application has real-time constraints so the GC delays > cannot be tolerated. I selected those GC options _in order to avoid > _long GC times. > > Additionally, these periods coincide with high CPU for that JVM > process. From 5-20% CPU where it is normally, it jumps to 60% ore more. > Once GC is done, our app rushes to catch up with tasks that had to wait > for GC to finish. > > Answering another question from a member who has kindly responded, yes > the server is running other stuff. Basically it runs three tomcats, the > main one being this one. It also runs a DB2 database that has > close-to-zero activity. > > George > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus >
