Re: JVM Optimizations

2016-06-10 Thread Barry Kaplan
Tom, Thanks, that's very good to know. What kind of instances EC2 instances are you using for your brokers? -barry On Fri, Jun 10, 2016 at 4:17 PM, Tom Crayford wrote: > Barry, > > No, because Kafka also relies heavily on the OS page cache, which uses > memory. You'd

Re: JVM Optimizations

2016-06-10 Thread Tom Crayford
Barry, No, because Kafka also relies heavily on the OS page cache, which uses memory. You'd roughly want to allocate enough page cache to hold all the messages for your consumers for say, 30s. Kafka also (in our experience on EC2) tends to run out of network far before it runs out of memory or

Re: JVM Optimizations

2016-06-10 Thread Barry Kaplan
If too much heap cause problems, would it make sense to run multiple brokers on a box with lots memory? For example, an EC2 D2 instance types has way way more ram than kafka could ever use - -but it has fast connected disks. Would running a broker per disk make sense in this case? -barry

Re: JVM Optimizations

2016-06-09 Thread Todd Palino
Xmx256M" > > > > > > > fi > > > > > > > > > > > > > > # JVM performance options > > > > > > > if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then > > > > > > > KAFKA_JVM_PERFORMANCE_OPTS="-server

Re: JVM Optimizations

2016-06-09 Thread Dustin Cote
; then > > > > > > KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC > > > > -XX:MaxGCPauseMillis=20 > > > > > > -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC > > > > > > -Djava.awt.headless=true" > > > &g

Re: JVM Optimizations

2016-06-09 Thread Shane Hender
XX:+DisableExplicitGC > > > > > -Djava.awt.headless=true" > > > > > fi > > > > > > > > > > > > > > > Is this the confluent doc you were referring to? > > > > > https://protect-us.mimecast.com/s/arXXB

Re: JVM Optimizations

2016-06-09 Thread Dustin Cote
; > > > > Thanks! > > > > > > > > Lawrence Weikum > > > > > > > > > > > > On 6/9/16, 1:32 PM, "Tom Crayford" <tcrayf...@heroku.com> wrote: > > > > > > > > >Hi Lawrence, > > > >

Re: JVM Optimizations

2016-06-09 Thread Stephen Powis
onfluent > docs > > > >on JVM tuning iirc. We simply use the G1 and a 4GB Max heap and things > > > work > > > >well (running many thousands of clusters). > > > > > > > >Thanks > > > >Tom Crayford > > > &g

Re: JVM Optimizations

2016-06-09 Thread Ben Osheroff
gt; >Tom Crayford > > >Heroku Kafka > > > > > >On Thursday, 9 June 2016, Lawrence Weikum <lwei...@pandora.com> wrote: > > > > > >> Hello all, > > >> > > >> We’ve been running a benchmark test on a Kafka cl

Re: JVM Optimizations

2016-06-09 Thread Stephen Powis
ce Weikum <lwei...@pandora.com> wrote: > > > >> Hello all, > >> > >> We’ve been running a benchmark test on a Kafka cluster of ours running > >> 0.9.0.1 – slamming it with messages to see when/if things might break. > >> During our test, we

Re: JVM Optimizations

2016-06-09 Thread Lawrence Weikum
0.9.0.1 – slamming it with messages to see when/if things might break. >> During our test, we caused two brokers to throw OutOfMemory errors (looks >> like from the Heap) even though each machine still has 43% of the total >> memory unused. >> >> I’m curious what JVM o

Re: JVM Optimizations

2016-06-09 Thread Tom Crayford
even though each machine still has 43% of the total > memory unused. > > I’m curious what JVM optimizations are recommended for Kafka brokers? Or > if there aren’t any that are recommended, what are some optimizations > others are using to keep the brokers running smoothly? > > Best, > > Lawrence Weikum > >

JVM Optimizations

2016-06-09 Thread Lawrence Weikum
% of the total memory unused. I’m curious what JVM optimizations are recommended for Kafka brokers? Or if there aren’t any that are recommended, what are some optimizations others are using to keep the brokers running smoothly? Best, Lawrence Weikum