Hi Chris, I think the first step would be to set up the system so we can easily identify the bottlenecks. With your setup I'm currently worried about 2 things:
1. the system is not being driven hard enough. In particular 1 producer might not be enough. I'd recommend running 3 producer processes. 2. CPU interference between producers and Kafka itself, since they are on the same box. Given that you have 10Gbps you should probably set up the producer on a different machine to Kafka (so a 3 machine setup: producer, zk, 1 kafka broker). With these two in place, the next thing to look at would be perf metrics to see if CPU, network and storage is the bottleneck. Would you have access to those metrics while the test is running. Also, for calibration, I'd run a test with small (100 byte) values. That way we can compare to https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines <https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines> for example. Does this help? Thanks Eno > On 9 Oct 2016, at 18:19, Christopher Stelly <[email protected]> wrote: > > Hey Eno, thanks > > Yes, localhost would mean testing SSD, maybe moving to a remote producer in > the future over 10GBe but baby steps first. > > My record size is pretty flexible - I've tested from 4k to 16M and the > results are +- 10MB/s. > > SSD is a samsung 850, claiming 100k IOPS and >500MB/s read/write. > > Again, thanks! > > On Sun, Oct 9, 2016 at 12:55 PM, Eno Thereska <[email protected]> > wrote: > >> Hi Chris, >> >> A couple of things: looks like you're primarily testing the SSD if you're >> running on the localhost, right? The 10GBe shouldn't matter in this case. >> >> The performance will depend a lot on the record sizes you are using. To >> get a ballpark number if would help to know the SSD type, would you be >> willing to share it? We can take it from there. >> >> Thanks >> Eno >> >>> On 9 Oct 2016, at 17:28, Christopher Stelly <[email protected]> wrote: >>> >>> Hello, >>> >>> The last thread available regarding 10GBe is about 2 years old, with no >>> obvious recommendations on tuning. >>> >>> Is there a more complex tuning guide than the example production config >>> available on Kafka's main site? Anything other than the list of possible >>> configs? >>> >>> I currently have access to a rather substantial academic cluster to test >>> on, including multiple machines with the following hardware: >>> >>> 10GBe NICs >>> 250GB RAM each >>> SSDs on each >>> (also, optional access to single NVMe) >>> >>> Using Golang's Kafka interface, I can only seem to get about 80MB/s on >> the >>> producer pushing to logs on the localhost, using no replication and >> reading >>> from/logging to SSD. If it helps, I can post my configs. I've tried >>> fiddling with a bunch of broker configs as well as producer configs, >>> raising the memory limits, max message size, io&network threads etc. >>> >>> Since the last post from 2014 indicates that there is no public >>> benchmarking for 10GBe, I'd be happy to run benchmarks /publish results >> on >>> this hardware if we can get it tuned up properly. >>> >>> What kind of broker/producer/consumer settings would you recommend? >>> >>> Thanks! >>> - chris >> >>
