We got 300ms for produce-localtimems .999th percentile for some machines of one cluster, actually the value for other machines of the cluster at most around 100ms. The QPS only 20~80 per second. The strange things is that only several machines has this kind of issue, all other machines has no this kind of issue. And the 99th percentile also less than 80ms. And the disk queue length less than 0.2 We're run on windows (after apply fix of kafka-1646) and we enable JBOD.
We dump the JMX counters every two minutes, here is some number: 1,04/23/2015 00:00:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,232.00 1,04/23/2015 00:02:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,33 1,04/23/2015 00:04:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,73 1,04/23/2015 00:06:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,98 1,04/23/2015 00:08:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,188.00 1,04/23/2015 00:10:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,93 1,04/23/2015 00:12:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,156.00 1,04/23/2015 00:14:00, kafka.network.requestmetrics.produce-localtimems.999thpercentile,300.00 1,04/23/2015 00:16:00, kafka.network.requestmetrics.produce-localtimems.999thpercentile,37 Thanks, Honghai
