-Original Message-
From: Guozhang Wang [mailto:wangg...@gmail.com]
Sent: 14 May 2015 18:57
To: users@kafka.apache.org
Subject: Re: Experiences testing new producer performance across multiple
threads/producer counts
Regarding the issue that adding more partitions kill the performance: I
would
re-visit this and do deeper profiling over the next week and
will revert here with findings.
Regards
Garry
-Original Message-
From: Guozhang Wang [mailto:wangg...@gmail.com]
Sent: 14 May 2015 18:57
To: users@kafka.apache.org
Subject: Re: Experiences testing new producer performance
Regarding the issue that adding more partitions kill the performance: I
would suspect it maybe due to not-sufficient batching. Note that in the new
producer batching is done per-partition, and if linger.ms setting low,
partition data may not be batched enough before they got sent to the
brokers.
Hi,
I talked with Gwen at Strata last week and promised to share some of my
experiences benchmarking an app reliant on the new producer. I'm using
relatively meaty boxes running my producer code (24 core/64GB RAM) but I wasn't
pushing them until I got them on the same 10GB fabric as the Kafka
Thanks for sharing this, Garry. I actually did similar tests before but
unfortunately lost the test data because my laptop rebooted and I forgot
to save the dataŠ
Anyway, several things to verify:
1. Remember KafkaProducer holds lock per partition. So if you have only
one partition in the target
Hey Garry,
Super interesting. We honestly never did a ton of performance tuning on the
producer. I checked the profiles early on in development and we fixed a few
issues that popped up in deployment, but I don't think anyone has done a
really scientific look. If you (or anyone else) want to dive
