One thing that changed in 0.9.0.0 is the introduction of `max.block.ms`, which is 60 seconds by default. Since you have "block.on.buffer.full=false", you may want to tweak `max.block.ms`.
Ismael On Mon, Sep 5, 2016 at 1:54 PM, Ismael Juma <[email protected]> wrote: > Hi Yifan, > > Just to make sure I understand, the comparison is between 0.8.2.1 and > 0.10.0.0 clients (both) connecting to a 0.10.0.0 cluster? We haven't > received reports of increased average producer latency and I'm not aware of > a change that could cause such a dramatic difference. Is compression being > used and did you paste the full producer config? > > Ismael > > On Fri, Sep 2, 2016 at 6:31 PM, Yifan Ying <[email protected]> wrote: > >> The load before and after the upgrade are pretty similar. And all these >> configs stayed same too. >> >> Also please correct me if I am wrong. I think the request latency = >> response received time - request created time. And request created time is >> created when the records are ready to be sent after 'lingering'. >> >> Yifan >> >> On Thu, Sep 1, 2016 at 11:01 PM, Gerard Klijs <[email protected]> >> wrote: >> >> > With a linger of 5 seconds, 2-3 seconds would make sense when the load >> is >> > smaller, are are sure the measurements with 0.8.2.1 where with the same >> > load and/or linger worked correctly there? >> > >> > On Fri, Sep 2, 2016 at 1:12 AM Yifan Ying <[email protected]> wrote: >> > >> > > We tried to upgrade the Kafka clients dependency from 0.8.2.1 to >> > 0.10.0.0. >> > > As I understand, the producer client doesn't have major changes in >> 0.10, >> > so >> > > we kept the same producer config in the upgrade: >> > > >> > > retries 3 >> > > retry.backoff.ms 5000 >> > > timeout.ms 10000 >> > > block.on.buffer.full false >> > > linger.ms 5000 >> > > metadata.fetch.timeout.ms 30000 >> > > >> > > However, after the upgrade, the avg producer request latency >> increased by >> > > 5×, from a few hundreds milliseconds to 2-3 seconds. Anyone has seen >> this >> > > issue? Is there any change in 0.9/0.10 that would cause this? >> > > >> > > Thanks! >> > > >> > > -- >> > > Yifan >> > > >> > >> >> >> >> -- >> Yifan >> > >
