Yeah this will involve some experimentation. The metrics are visible with jconsole or another jmx viewer.
It may also be worth looking at the cpu usage per-thread (e.g. start top and press 't' I think). Another simple test for broker vs client as the bottleneck is just to start another producer or consumer and see if that improves throughput (if so it is probably a client bottleneck). -Jay On Tue, Nov 18, 2014 at 4:44 PM, Manu Zhang <owenzhang1...@gmail.com> wrote: > Thanks Jay for the quick response. > > Yes, it's a single producer and consumer both configured with multiple > threads but I'm not using the new producer. > CPU is typically 50% utilized on client and merely used on broker. Disks > aren't busy either as a lot of data are cached in memory. > Would you please give a link for the producer metrics you are referring to > ? > > Thanks, > Manu > > On Wed, Nov 19, 2014 at 2:39 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > Hey Manu, > > > > I'm not aware of a benchmark on 10GbE. I'd love to see that though. > Diving > > into the results may help us find bottlenecks hidden by the slower > network. > > > > Can you figure out where the bottleneck is in your test? I assume this > is a > > single producer and consumer instance and you are using the new producer > as > > in those benchmarks? > > > > This can be slightly tricky as it can be cpu or I/O on either the clients > > or the brokers. You basically have to look at top, iostat, and the jmx > > metrics for clues. The producer has good metrics that explain whether it > is > > spending most of its time waiting or sending data. Not sure if there is a > > similar diagnostic for the consumer. > > > > -Jay > > > > On Tue, Nov 18, 2014 at 5:10 AM, Manu Zhang <owenzhang1...@gmail.com> > > wrote: > > > > > Hi all, > > > > > > I have been trying out kafka benchmarks described in Jay's > > > benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > > < > > > > > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machine > > > >. > > > I'm able to get similar results on a 4-node GbE network whose in-bytes > > > could be saturated at 120MB/s. However, on a 4-node, 10GbE network, I > can > > > not get in-bytes higher than 150MB/s. *Has anyone benchmarked kafka on > a > > > 10GbE network ? Any rule of thumb on 10GbE network for configurations > of > > > broker, producer and consumer ? * > > > > > > My kafka version is 0.8.1.1 and I've created a topic with 8 partitions > > with > > > 1 replica distributed evenly among the 4 nodes. Message size is 100 > > bytes. > > > I use all the default kafka settings. > > > My cluster has 4 nodes, where each node has 32 cores, 128MB RAM and 3 > > disks > > > for kafka. > > > > > > I've tried increasing message size to 1000 bytes which improved > > producer's > > > throughput but not consumer's. > > > > > > > > > Thanks, > > > Manu > > > > > >
