Tom,
Thanks, that's very good to know. What kind of instances EC2 instances are
you using for your brokers?
-barry
On Fri, Jun 10, 2016 at 4:17 PM, Tom Crayford wrote:
> Barry,
>
> No, because Kafka also relies heavily on the OS page cache, which uses
> memory. You'd
Barry,
No, because Kafka also relies heavily on the OS page cache, which uses
memory. You'd roughly want to allocate enough page cache to hold all the
messages for your consumers for say, 30s.
Kafka also (in our experience on EC2) tends to run out of network far
before it runs out of memory or
If too much heap cause problems, would it make sense to run multiple
brokers on a box with lots memory? For example, an EC2 D2 instance types
has way way more ram than kafka could ever use - -but it has fast connected
disks.
Would running a broker per disk make sense in this case?
-barry
Xmx256M"
> > > > > > > fi
> > > > > > >
> > > > > > > # JVM performance options
> > > > > > > if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
> > > > > > > KAFKA_JVM_PERFORMANCE_OPTS="-server
; then
> > > > > > KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC
> > > > -XX:MaxGCPauseMillis=20
> > > > > > -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC
> > > > > > -Djava.awt.headless=true"
> > > &g
XX:+DisableExplicitGC
> > > > > -Djava.awt.headless=true"
> > > > > fi
> > > > >
> > > > >
> > > > > Is this the confluent doc you were referring to?
> > > > > https://protect-us.mimecast.com/s/arXXB
;
> > > > Thanks!
> > > >
> > > > Lawrence Weikum
> > > >
> > > >
> > > > On 6/9/16, 1:32 PM, "Tom Crayford" <tcrayf...@heroku.com> wrote:
> > > >
> > > > >Hi Lawrence,
> > > >
onfluent
> docs
> > > >on JVM tuning iirc. We simply use the G1 and a 4GB Max heap and things
> > > work
> > > >well (running many thousands of clusters).
> > > >
> > > >Thanks
> > > >Tom Crayford
> > > &g
gt; >Tom Crayford
> > >Heroku Kafka
> > >
> > >On Thursday, 9 June 2016, Lawrence Weikum <lwei...@pandora.com> wrote:
> > >
> > >> Hello all,
> > >>
> > >> We’ve been running a benchmark test on a Kafka cl
ce Weikum <lwei...@pandora.com> wrote:
> >
> >> Hello all,
> >>
> >> We’ve been running a benchmark test on a Kafka cluster of ours running
> >> 0.9.0.1 – slamming it with messages to see when/if things might break.
> >> During our test, we
0.9.0.1 – slamming it with messages to see when/if things might break.
>> During our test, we caused two brokers to throw OutOfMemory errors (looks
>> like from the Heap) even though each machine still has 43% of the total
>> memory unused.
>>
>> I’m curious what JVM o
even though each machine still has 43% of the total
> memory unused.
>
> I’m curious what JVM optimizations are recommended for Kafka brokers? Or
> if there aren’t any that are recommended, what are some optimizations
> others are using to keep the brokers running smoothly?
>
> Best,
>
> Lawrence Weikum
>
>
% of the total memory unused.
I’m curious what JVM optimizations are recommended for Kafka brokers? Or if
there aren’t any that are recommended, what are some optimizations others are
using to keep the brokers running smoothly?
Best,
Lawrence Weikum
13 matches
Mail list logo