Thanks for sharing this, Garry. I actually did similar tests before but unfortunately lost the test data because my laptop rebooted and I forgot to save the dataŠ
Anyway, several things to verify: 1. Remember KafkaProducer holds lock per partition. So if you have only one partition in the target topic and many application threads. Lock contention could be an issue. 2. It matters that how frequent the sender thread wake up and runs. You can take a look at the following sensors to further verify whether the sender thread really become a bottleneck or not: Select-rate Io-wait-time-ns-avg Io-time-ns-avg 3. Batch size matters, so take a look at the sensor batch-size-avg and see if the average batch size makes sense or not. Looking forward to your further profiling. My thinking is that unless you are sending very small messages to a small number of partitions. You don¹t need to worry about to use more than one producer. Thanks. Jiangjie (Becket) Qin On 5/13/15, 2:40 PM, "Garry Turkington" <g.turking...@improvedigital.com> wrote: >Hi, > >I talked with Gwen at Strata last week and promised to share some of my >experiences benchmarking an app reliant on the new producer. I'm using >relatively meaty boxes running my producer code (24 core/64GB RAM) but I >wasn't pushing them until I got them on the same 10GB fabric as the Kafka >cluster they are using (saturating the prior 1GB NICs was just too easy). >There are 5 brokers, 24 core/192GB RAM/8*2TB disks, running 0.8.2.1. > >With lots of cores and a dedicated box the question was then how to >deploy my application. In particular how many worker threads and how many >instances of the KafkaProducer to share amongst them. I also wanted to >see how things would change as I scale up the thread count. > >I ripped out the data retrieval part of my app (it reads from S3) and >instead replaced it with some code to produce random records of average >size 500 bytes but varying between 250 and 750. I started the app >running, ignored the first 25m messages then measured the timing for the >next 100m and calculated the average messages/sec written to Kafka >across that run. > >Starting small I created 4 application threads with a range of approaches >to sharing KafkaProducer instances. The records written to the Kafka >cluster per second were as follows: > >4 threads all sharing 1 client: 332K >4 threads sharing 2 clients: 333K >4 threads, dedicated client per thread: 310K > >Note that when I had 2 KafkaProducer clients as in the second line above >each was used by 2 threads. Similar below, number of threads/number of >clients is the max number of threads per KafkaProducer instance. > >As can be seen from the above there's not much in it. Scaling up to 8 >application threads the numbers looked like: > >8 threads sharing 1 client: 387K >8 threads sharing 2 clients: 609K >8 threads sharing 4 clients: 628K >8 threads with dedicated client per thread: 527K > >This time sharing a single producer client across all threads has by far >the worse performance and isn't much better than when using 4 >application threads. The 2 and 4 client options are much better and are >in the ballpark of 2x the 4 thread performance. A dedicated client per >thread isn't quite as good but isn't so far off to be unusable. So then >taking it to 16 application threads: > >16 threads sharing 1 client: 380K >16 threads sharing 2 clients: 675K >16 threads sharing 4 clients: 869K >16 threads sharing 8 clients: 733K >16 threads with a dedicated client per thread: 491K > >This gives a much clearer performance curve. The 16 thread/4 producer >client is by far the best performance but it is still far from 4x the >4-thread or 2x the 8-thread mark. At this point I seem to be hitting some >limiting factor. On the client machine memory was still lightly used, >network was peaking just over 4GB/sec but CPU load was showing 1 minute >load averages around 18-20. CPU load did seem to increase with as did the >number of KafkaProducer instances but that is more a conclusion from >memory and not backed by hard numbers. > >For completeness sake I did do a 24 thread test but the numbers are as >you'd expect. 1 client and 24 both showed poor performance. 4,6 or 8 >clients (24 has more ways of dividing it by 2!) all showed performance >around that of the 16 thread/4 client run above. The other configs were >in-between. > >With my full application I've found the best deployment so far is to have > multiple instances running on the same box. I can get much better >performance from 3 instances each with 8 threads than 1 instance with 24 >threads. This is almost certainly because when adding in my own >application logic and the AWS clients there is just a lot more contention >- not to mention much more i/o waits -- in each application instance. The >benchmark variant doesn't have as much happening but just to compare I >ran a few concurrent instances: > >2 copies of 8 threads sharing 4 clients: 780K total >2 copies of 8 threads sharing 2 clients: 870K total >3 copies of 8 threads sharing 2 clients: 945k total > >So bottom line - around 900K/sec is the max I can get from one of these >hosts for my application. At which point I brought a 2nd host to bear and >ran 2 concurrent instances of the best performing config on each: > >2 copies of 16 threads sharing 4 clients on 2 hosts: 1458k total > >This doesn't quite give 2x the single box performance but it does show >that the cluster has capacity to spare beyond what the single client host >can drive. This was also backed up by the metrics on the brokers, they >got busy but moderately so given the amount of work they were doing. > >At this point things did get a bit 'ragged edge'. I noticed a very high >rate of ISR churn on the brokers, it looked like the replicas were having >trouble keeping up with the master and hosts were constantly being >dropped out then re-added to the ISR. I had set the test topic to have a >relatively low partition count (1 per spindle) so I doubled that to see >if it could help the ISRs remain stable. And my performance fell through >the floor. So whereas I thought this was an equation involving >application threads and producer instances perhaps partition count is a >third. I need look into that some more but so far it looks like that for >my application - I'm not suggesting this is a universal truth -- sharing >a KafkaProducer instance amongst around 4 threads is the sweet spot. > >I'll be doing further profiling of my application so I'll flag to the >list anything that appears within the producer itself. And because 900K >messages/sec was so close to a significant number I modified my code that >generates the messages to keep the key random for each message but to use >repeated message bodies across multiple messages. At which point 1.05m >messages/sec was possible - from a single box. Nice. :) > >This turned out much longer than planned, I probably should have blogged >this somewhere. If anyone reads this far hope it is useful or of >interest, I'd be interested in hearing if the profiles I'm seeing are >expected and if any other tests would be useful. > >Regards >Garry
