My current theory, which I haven't dug into the source to confirm, is that said buffers are being pre-allocated. Because the kafka instance is relatively bored, they end up living long enough to see a few collections and be promoted. I could be way off base though.
Command line, broken out for a little better readability: 18918 /opt/java/1.8.0_40/bin/java -cp :/mnt/services/kafka08/etc::/mnt/services/kafka08/current/lib/java:/mnt/services/kafka08/current/lib/java/zookeeper-3.4.6.jar:/mnt/services/kafka08/current/lib/java/scala-library-2.11.5.jar:/mnt/services/kafka08/current/lib/java/lz4-1.2.0.jar:/mnt/services/kafka08/current/lib/java/scala-parser-combinators_2.11-1.0.2.jar:/mnt/services/kafka08/current/lib/java/metrics-core-2.2.0.jar:/mnt/services/kafka08/current/lib/java/kafka_2.11-0.8.2.0-scaladoc.jar:/mnt/services/kafka08/current/lib/java/kafka_2.11-0.8.2.0-javadoc.jar:/mnt/services/kafka08/current/lib/java/scala-xml_2.11-1.0.2.jar:/mnt/services/kafka08/current/lib/java/kafka_2.11-0.8.2.0.jar:/mnt/services/kafka08/current/lib/java/kafka-clients-0.8.2.0.jar:/mnt/services/kafka08/current/lib/java/snappy-java-1.1.1.6.jar:/mnt/services/kafka08/current/lib/java/kafka_2.11-0.8.2.0-test.jar:/mnt/services/kafka08/current/lib/java/log4j-1.2.16.jar:/mnt/services/kafka08/current/lib/java/kafka_2.11-0.8.2.0-sources.jar:/mnt/services/kafka08/current/lib/java/slf4j-api-1.7.6.jar:/mnt/services/kafka08/current/lib/java/slf4j-log4j12-1.6.1.jar:/mnt/services/kafka08/current/lib/java/zkclient-0.3.jar:/mnt/services/kafka08/current/lib/java/jopt-simple-3.2.jar: -Xms4096M -Xmx4096M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/services/kafka08/var/dump/2015-10-13-17_34_17.hprof -XX:+PrintAdaptiveSizePolicy -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -Xloggc:/mnt/services/kafka08/var/log/kafka08-gc-2015-10-13-17_34_17.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Dlog4j.configuration=file:///mnt/services/kafka08/etc/log4j.properties kafka.Kafka /mnt/services/kafka08/etc/kafka08.properties On Wed, Oct 14, 2015 at 11:37 AM, Todd Palino <tpal...@gmail.com> wrote: > We've had no problems with G1 in all of our clusters with varying load > levels. I think we've seen an occasional long GC here and there, but > nothing recurring at this point. > > What's the full command line that you're using with all the options? > > -Todd > > > On Wed, Oct 14, 2015 at 2:18 PM, Scott Clasen <sc...@heroku.com> wrote: > > > You can also use -Xmn with that gc to size the new gen such that those > > buffers don't get tenured > > > > I don't think that's an option with G1 > > > > On Wednesday, October 14, 2015, Cory Kolbeck <ckolb...@gmail.com> wrote: > > > > > I'm not sure that will help here, you'll likely have the same > > > medium-lifetime buffers getting into the tenured generation and forcing > > > large collections. > > > > > > On Wed, Oct 14, 2015 at 10:00 AM, Gerrit Jansen van Vuuren < > > > gerrit...@gmail.com <javascript:;>> wrote: > > > > > > > Hi, > > > > > > > > I've seen pauses using G1 in other applications and have found that > > > > -XX:+UseParallelGC > > > > -XX:+UseParallelOldGC works best if you're having GC issues in > general > > > on > > > > the JVM. > > > > > > > > > > > > Regards, > > > > Gerrit > > > > > > > > On Wed, Oct 14, 2015 at 4:28 PM, Cory Kolbeck <ckolb...@gmail.com > > > <javascript:;>> wrote: > > > > > > > > > Hi folks, > > > > > > > > > > I'm a bit new to the operational side of G1, but pretty familiar > with > > > its > > > > > basic concept. We recently set up a Kafka cluster to support a new > > > > product, > > > > > and are seeing some suboptimal GC performance. We're using the > > > parameters > > > > > suggested in the docs, except for having switched to java 1.8_40 in > > > order > > > > > to get better memory debugging. Even though the cluster is handling > > > only > > > > > 2-3k messages per second per node, we see periodic 11-18 second > > > > > stop-the-world pauses on a roughly hourly cadence. I've turned on > > > > > additional GC logging, and see no humongous allocations, it all > seems > > > to > > > > be > > > > > buffers making it into the tenured gen. They appear to be > > collectable, > > > as > > > > > the collection triggered by dumping the heap collects them all. > Ideas > > > for > > > > > additional diagnosis or tuning very welcome. > > > > > > > > > > --Cory > > > > > > > > > > > > > > >
