Neha,
I got some time today to test the JMX values you mentioned. My test setup
includes a Kafka broker, a PHP based load generator that generates messages
at a steady rate, and a Storm Kafka consumer. The PHP load generator
continuously pumps in a standard 3KB message with a sleep interval of 100
Thanks Neha and Jun for the pointers. We will try and evaluate this as well.
Hemanth
On Sat, Sep 13, 2014 at 4:45 AM, Neha Narkhede
wrote:
> Hemanth,
>
> Specifically, you'd want to monitor
> kafka:type=kafka.SocketServerStats:getMaxProduceRequestMs and
> kafka:type=kafka.LogFlushStats:getMaxFl
Hemanth,
Specifically, you'd want to monitor
kafka:type=kafka.SocketServerStats:getMaxProduceRequestMs and
kafka:type=kafka.LogFlushStats:getMaxFlushMs. If the broker is under load
due to frequent flushes, it will almost certainly show up as spikes in the
flush latency and consequently the produce
One of the differences btw 0.7.x and 0.8.x is that the latter does the I/O
flushing in the background. So, in 0.7.x, more frequent I/O flushing will
increase the producer latency.
Thanks,
Jun
On Thu, Sep 11, 2014 at 5:48 PM, Hemanth Yamijala
wrote:
> Neha,
>
> Thanks. We are on 0.7.2. I have w
Neha,
Thanks. We are on 0.7.2. I have written on another thread on the list here
about one of the reasons we are stuck - the absence of a PHP client for our
front end producer systems. (On a side note, would appreciate if any inputs
can be given on that thread as well)
When you mean performance,
I should mention that the impact of doing so is much higher wrt to taking a
hit on performance, on versions < 0.8.1. As long as you're on 0.8.1 or
later, it should mostly be fine. You might want to keep a close tab on how
your iostat numbers are doing, to be sure.
On Wed, Sep 10, 2014 at 5:46 PM,
Thanks Jun.
On Thu, Sep 11, 2014 at 4:13 AM, Jun Rao wrote:
> As long as the I/O load is reasonable, this is probably ok.
>
> Thanks,
>
> Jun
>
> On Wed, Sep 10, 2014 at 4:59 AM, Hemanth Yamijala
> wrote:
>
> > Hi folks,
> >
> > In order to meet latency requirements for a system we are building
As long as the I/O load is reasonable, this is probably ok.
Thanks,
Jun
On Wed, Sep 10, 2014 at 4:59 AM, Hemanth Yamijala
wrote:
> Hi folks,
>
> In order to meet latency requirements for a system we are building, we
> tested with different values of the above two parameters and found that
> se
Hi folks,
In order to meet latency requirements for a system we are building, we
tested with different values of the above two parameters and found that
settings as low as 100 work best for us, balancing the required throughput
and latencies.
I just wanted to check if 100 is a sane value, notwith
