Hey Eno, thanks Yes, localhost would mean testing SSD, maybe moving to a remote producer in the future over 10GBe but baby steps first.
My record size is pretty flexible - I've tested from 4k to 16M and the results are +- 10MB/s. SSD is a samsung 850, claiming 100k IOPS and >500MB/s read/write. Again, thanks! On Sun, Oct 9, 2016 at 12:55 PM, Eno Thereska <eno.there...@gmail.com> wrote: > Hi Chris, > > A couple of things: looks like you're primarily testing the SSD if you're > running on the localhost, right? The 10GBe shouldn't matter in this case. > > The performance will depend a lot on the record sizes you are using. To > get a ballpark number if would help to know the SSD type, would you be > willing to share it? We can take it from there. > > Thanks > Eno > > > On 9 Oct 2016, at 17:28, Christopher Stelly <cdste...@gmail.com> wrote: > > > > Hello, > > > > The last thread available regarding 10GBe is about 2 years old, with no > > obvious recommendations on tuning. > > > > Is there a more complex tuning guide than the example production config > > available on Kafka's main site? Anything other than the list of possible > > configs? > > > > I currently have access to a rather substantial academic cluster to test > > on, including multiple machines with the following hardware: > > > > 10GBe NICs > > 250GB RAM each > > SSDs on each > > (also, optional access to single NVMe) > > > > Using Golang's Kafka interface, I can only seem to get about 80MB/s on > the > > producer pushing to logs on the localhost, using no replication and > reading > > from/logging to SSD. If it helps, I can post my configs. I've tried > > fiddling with a bunch of broker configs as well as producer configs, > > raising the memory limits, max message size, io&network threads etc. > > > > Since the last post from 2014 indicates that there is no public > > benchmarking for 10GBe, I'd be happy to run benchmarks /publish results > on > > this hardware if we can get it tuned up properly. > > > > What kind of broker/producer/consumer settings would you recommend? > > > > Thanks! > > - chris > >
