Re: Tuning for high RAM and 10GBe

Sounds good that you got up to 500MB/s. At that point I suspect you reach a sort of steady state where the cache is continuously flushing to the SSDs, so you are effectively bottlenecked by the SSD. I believe this is as expected (the bottleneck resource will dominate the end to end throughput

Re: Tuning for high RAM and 10GBe

With that link I came across the producer-perf-test tool, quite useful as it gets rid of the Go (Sarama) variable. Since it can quickly tweak settings, it's extremely useful. As you suggested Eno, I attempted to copy the LinkedIn settings. With 100 byte records, I get up to about 600,000

Re: Tuning for high RAM and 10GBe

Sure, good ideas. I'll try multiple producers, localhost and LAN, to see if any difference Yep, Gwen, the Sarama client. Anything to worry about there outside of setting the producer configs (which would you set?) and number of buffered channels? (currently, buffered channels up to 10k). Thanks!

Re: Tuning for high RAM and 10GBe

Out of curiosity - what is "Golang's Kafka interface"? Are you referring to Sarama client? On Sun, Oct 9, 2016 at 9:28 AM, Christopher Stelly wrote: > Hello, > > The last thread available regarding 10GBe is about 2 years old, with no > obvious recommendations on tuning. > >

Re: Tuning for high RAM and 10GBe

Hi Chris, I think the first step would be to set up the system so we can easily identify the bottlenecks. With your setup I'm currently worried about 2 things: 1. the system is not being driven hard enough. In particular 1 producer might not be enough. I'd recommend running 3 producer

Re: Tuning for high RAM and 10GBe

Hey Eno, thanks Yes, localhost would mean testing SSD, maybe moving to a remote producer in the future over 10GBe but baby steps first. My record size is pretty flexible - I've tested from 4k to 16M and the results are +- 10MB/s. SSD is a samsung 850, claiming 100k IOPS and >500MB/s read/write.

Re: Tuning for high RAM and 10GBe

Hi Chris, A couple of things: looks like you're primarily testing the SSD if you're running on the localhost, right? The 10GBe shouldn't matter in this case. The performance will depend a lot on the record sizes you are using. To get a ballpark number if would help to know the SSD type, would

Tuning for high RAM and 10GBe

Hello, The last thread available regarding 10GBe is about 2 years old, with no obvious recommendations on tuning. Is there a more complex tuning guide than the example production config available on Kafka's main site? Anything other than the list of possible configs? I currently have access to