Sure, good ideas. I'll try multiple producers, localhost and LAN, to see if any difference
Yep, Gwen, the Sarama client. Anything to worry about there outside of setting the producer configs (which would you set?) and number of buffered channels? (currently, buffered channels up to 10k). Thanks! On Mon, Oct 10, 2016 at 12:04 PM, Gwen Shapira <g...@confluent.io> wrote: > Out of curiosity - what is "Golang's Kafka interface"? Are you > referring to Sarama client? > > On Sun, Oct 9, 2016 at 9:28 AM, Christopher Stelly <cdste...@gmail.com> > wrote: > > Hello, > > > > The last thread available regarding 10GBe is about 2 years old, with no > > obvious recommendations on tuning. > > > > Is there a more complex tuning guide than the example production config > > available on Kafka's main site? Anything other than the list of possible > > configs? > > > > I currently have access to a rather substantial academic cluster to test > > on, including multiple machines with the following hardware: > > > > 10GBe NICs > > 250GB RAM each > > SSDs on each > > (also, optional access to single NVMe) > > > > Using Golang's Kafka interface, I can only seem to get about 80MB/s on > the > > producer pushing to logs on the localhost, using no replication and > reading > > from/logging to SSD. If it helps, I can post my configs. I've tried > > fiddling with a bunch of broker configs as well as producer configs, > > raising the memory limits, max message size, io&network threads etc. > > > > Since the last post from 2014 indicates that there is no public > > benchmarking for 10GBe, I'd be happy to run benchmarks /publish results > on > > this hardware if we can get it tuned up properly. > > > > What kind of broker/producer/consumer settings would you recommend? > > > > Thanks! > > - chris > > > > -- > Gwen Shapira > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog >
