Hi Ismael,

Thanks for replying.

Yes. It's the comparison between 0.8.2.1 and 0.10.0.0 clients connecting to
0.10.0.0 brokers(log.message.format.version=0.8.2.1). No compression is
used and that's all the producer config.

>From the Kafka code, I think the request-latency-avg metric seem not
include the blocking time, but only measures the time between creating
requests and getting response from brokers. Blocking on full buffer happens
before that. Please correct me if I am wrong.

Yifan


On Mon, Sep 5, 2016 at 5:59 AM, Ismael Juma <[email protected]> wrote:

> One thing that changed in 0.9.0.0 is the introduction of `max.block.ms`,
> which is 60 seconds by default. Since you have
> "block.on.buffer.full=false",
> you may want to tweak `max.block.ms`.
>
> Ismael
>
> On Mon, Sep 5, 2016 at 1:54 PM, Ismael Juma <[email protected]> wrote:
>
> > Hi Yifan,
> >
> > Just to make sure I understand, the comparison is between 0.8.2.1 and
> > 0.10.0.0 clients (both) connecting to a 0.10.0.0 cluster? We haven't
> > received reports of increased average producer latency and I'm not aware
> of
> > a change that could cause such a dramatic difference. Is compression
> being
> > used and did you paste the full producer config?
> >
> > Ismael
> >
> > On Fri, Sep 2, 2016 at 6:31 PM, Yifan Ying <[email protected]> wrote:
> >
> >> The load before and after the upgrade are pretty similar. And all these
> >> configs stayed same too.
> >>
> >> Also please correct me if I am wrong. I think the request latency =
> >> response received time - request created time. And request created time
> is
> >> created when the records are ready to be sent after 'lingering'.
> >>
> >> Yifan
> >>
> >> On Thu, Sep 1, 2016 at 11:01 PM, Gerard Klijs <[email protected]>
> >> wrote:
> >>
> >> > With a linger of 5 seconds, 2-3 seconds would make sense when the load
> >> is
> >> > smaller, are are sure the measurements with 0.8.2.1 where with the
> same
> >> > load and/or linger worked correctly there?
> >> >
> >> > On Fri, Sep 2, 2016 at 1:12 AM Yifan Ying <[email protected]> wrote:
> >> >
> >> > > We tried to upgrade the Kafka clients dependency from 0.8.2.1 to
> >> > 0.10.0.0.
> >> > > As I understand, the producer client doesn't have major changes in
> >> 0.10,
> >> > so
> >> > > we kept the same producer config in the upgrade:
> >> > >
> >> > > retries 3
> >> > > retry.backoff.ms 5000
> >> > > timeout.ms 10000
> >> > > block.on.buffer.full false
> >> > > linger.ms 5000
> >> > > metadata.fetch.timeout.ms 30000
> >> > >
> >> > > However, after the upgrade, the avg producer request latency
> >> increased by
> >> > > 5×, from a few hundreds milliseconds to 2-3 seconds. Anyone has seen
> >> this
> >> > > issue? Is there any change in 0.9/0.10 that would cause this?
> >> > >
> >> > > Thanks!
> >> > >
> >> > > --
> >> > > Yifan
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Yifan
> >>
> >
> >
>



-- 
Yifan

Reply via email to