Hi Chris, A couple of things: looks like you're primarily testing the SSD if you're running on the localhost, right? The 10GBe shouldn't matter in this case.
The performance will depend a lot on the record sizes you are using. To get a ballpark number if would help to know the SSD type, would you be willing to share it? We can take it from there. Thanks Eno > On 9 Oct 2016, at 17:28, Christopher Stelly <cdste...@gmail.com> wrote: > > Hello, > > The last thread available regarding 10GBe is about 2 years old, with no > obvious recommendations on tuning. > > Is there a more complex tuning guide than the example production config > available on Kafka's main site? Anything other than the list of possible > configs? > > I currently have access to a rather substantial academic cluster to test > on, including multiple machines with the following hardware: > > 10GBe NICs > 250GB RAM each > SSDs on each > (also, optional access to single NVMe) > > Using Golang's Kafka interface, I can only seem to get about 80MB/s on the > producer pushing to logs on the localhost, using no replication and reading > from/logging to SSD. If it helps, I can post my configs. I've tried > fiddling with a bunch of broker configs as well as producer configs, > raising the memory limits, max message size, io&network threads etc. > > Since the last post from 2014 indicates that there is no public > benchmarking for 10GBe, I'd be happy to run benchmarks /publish results on > this hardware if we can get it tuned up properly. > > What kind of broker/producer/consumer settings would you recommend? > > Thanks! > - chris
