Hi Guozhang, batch-size-avg and request-size-avg looked similar after the upgrade.
Yifan On Tue, Sep 6, 2016 at 2:19 PM, Guozhang Wang <[email protected]> wrote: > Yifan, > > Did you look into the producer metrics and check the avg. batch size / > request size? Did they change significantly compared with 0.8.2.1 clients > (there should be similar metrics but with different metrics name)? > > Guozhang > > On Mon, Sep 5, 2016 at 11:17 PM, Yifan Ying <[email protected]> wrote: > > > Hi Ismael, > > > > Thanks for replying. > > > > Yes. It's the comparison between 0.8.2.1 and 0.10.0.0 clients connecting > to > > 0.10.0.0 brokers(log.message.format.version=0.8.2.1). No compression is > > used and that's all the producer config. > > > > From the Kafka code, I think the request-latency-avg metric seem not > > include the blocking time, but only measures the time between creating > > requests and getting response from brokers. Blocking on full buffer > happens > > before that. Please correct me if I am wrong. > > > > Yifan > > > > > > On Mon, Sep 5, 2016 at 5:59 AM, Ismael Juma <[email protected]> wrote: > > > > > One thing that changed in 0.9.0.0 is the introduction of `max.block.ms > `, > > > which is 60 seconds by default. Since you have > > > "block.on.buffer.full=false", > > > you may want to tweak `max.block.ms`. > > > > > > Ismael > > > > > > On Mon, Sep 5, 2016 at 1:54 PM, Ismael Juma <[email protected]> wrote: > > > > > > > Hi Yifan, > > > > > > > > Just to make sure I understand, the comparison is between 0.8.2.1 and > > > > 0.10.0.0 clients (both) connecting to a 0.10.0.0 cluster? We haven't > > > > received reports of increased average producer latency and I'm not > > aware > > > of > > > > a change that could cause such a dramatic difference. Is compression > > > being > > > > used and did you paste the full producer config? > > > > > > > > Ismael > > > > > > > > On Fri, Sep 2, 2016 at 6:31 PM, Yifan Ying <[email protected]> > wrote: > > > > > > > >> The load before and after the upgrade are pretty similar. And all > > these > > > >> configs stayed same too. > > > >> > > > >> Also please correct me if I am wrong. I think the request latency = > > > >> response received time - request created time. And request created > > time > > > is > > > >> created when the records are ready to be sent after 'lingering'. > > > >> > > > >> Yifan > > > >> > > > >> On Thu, Sep 1, 2016 at 11:01 PM, Gerard Klijs < > > [email protected]> > > > >> wrote: > > > >> > > > >> > With a linger of 5 seconds, 2-3 seconds would make sense when the > > load > > > >> is > > > >> > smaller, are are sure the measurements with 0.8.2.1 where with the > > > same > > > >> > load and/or linger worked correctly there? > > > >> > > > > >> > On Fri, Sep 2, 2016 at 1:12 AM Yifan Ying <[email protected]> > > wrote: > > > >> > > > > >> > > We tried to upgrade the Kafka clients dependency from 0.8.2.1 to > > > >> > 0.10.0.0. > > > >> > > As I understand, the producer client doesn't have major changes > in > > > >> 0.10, > > > >> > so > > > >> > > we kept the same producer config in the upgrade: > > > >> > > > > > >> > > retries 3 > > > >> > > retry.backoff.ms 5000 > > > >> > > timeout.ms 10000 > > > >> > > block.on.buffer.full false > > > >> > > linger.ms 5000 > > > >> > > metadata.fetch.timeout.ms 30000 > > > >> > > > > > >> > > However, after the upgrade, the avg producer request latency > > > >> increased by > > > >> > > 5×, from a few hundreds milliseconds to 2-3 seconds. Anyone has > > seen > > > >> this > > > >> > > issue? Is there any change in 0.9/0.10 that would cause this? > > > >> > > > > > >> > > Thanks! > > > >> > > > > > >> > > -- > > > >> > > Yifan > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> Yifan > > > >> > > > > > > > > > > > > > > > > > > > -- > > Yifan > > > > > > -- > -- Guozhang > -- Yifan
