Thanks. The results make sense. Higher consistency (ack=-1 and ack=2) typically means longer latency.
Do those number match our java producers? Thanks, Jun On Mon, Nov 25, 2013 at 1:16 PM, Magnus Edenhill <mag...@edenhill.se> wrote: > Producing to one partition, no replication, required.acks = 0: > % 1000000 messages and 100000000 bytes produced in 1215ms: 822383 msgs/s > and 82.24 Mb/s, 0 messages failed, no compression > > Producing to one partition, no replication, required.acks = -1: > % 1000000 messages and 100000000 bytes produced in 1422ms: 703091 msgs/s > and 70.31 Mb/s, 0 messages failed, no compression > > Producing to one partition, no replication, required.acks = 1: > % 1000000 messages and 100000000 bytes produced in 1295ms: 771881 msgs/s > and 77.19 Mb/s, 0 messages failed, no compression > > > > Producing to one partition, replication factor 2, 2 brokers ISR, > required.acks = 0: > % 1000000 messages and 100000000 bytes produced in 1354ms: 738483 msgs/s > and 73.85 Mb/s, 0 messages failed, no compression > > Producing to one partition, replication factor 2, 2 brokers ISR, > required.acks = -1: > % 1000000 messages and 100000000 bytes produced in 3698ms: 270396 msgs/s > and 27.04 Mb/s, 0 messages failed, no compression > > Producing to one partition, replication factor 2, 2 brokers ISR, > required.acks = 1: > % 1000000 messages and 100000000 bytes produced in 1360ms: 735224 msgs/s > and 73.52 Mb/s, 0 messages failed, no compression > > Producing to one partition, replication factor 2, 2 brokers ISR, > required.acks = 2: > % 1000000 messages and 100000000 bytes produced in 3568ms: 280241 msgs/s > and 28.02 Mb/s, 0 messages failed, no compression > > > These are the maximum values from a smaller number of naive tests. > > It would be interesting to see some numbers from relevant environments with > proper hardware and networks. > (rdkafka_performance -P -t <topic> -p <partition> -s <msgsize> -a > <required.acks> -c 1000000 -q) > > Regards, > Magnus > > > 2013/11/25 Jun Rao <jun...@gmail.com> > > > Thanks for sharing the results. Was the topic created with replication > > factor of 2? Could you test acks=-1 as well? > > > > Thanks, > > > > Jun > > > > > > On Mon, Nov 25, 2013 at 4:30 AM, Magnus Edenhill <mag...@edenhill.se> > > wrote: > > > > > The following tests were using a single producer application > > > (examples/rdkafka_performance): > > > > > > * Test1: 2 brokers, 2 partitions, required.acks=2, 100 byte messages: > > > 850000 messages/second, 85 MB/second > > > > > > * Test2: 1 broker, 1 partition, required.acks=0, 100 byte messages: > > 710000 > > > messages/second, 71 MB/second > > > > > > * Test3: 2 broker2, 2 partitions, required.acks=2, 100 byte messages, > > > snappy compression: 300000 messages/second, 30 MB/second > > > > > > * Test4: 2 broker2, 2 partitions, required.acks=2, 100 byte messages, > > gzip > > > compression: 230000 messages/second, 23 MB/second > > > > > > > > > log.flush broker configuration was increased to avoid the disk being > the > > > bottleneck. > > > > > > > > > /Magnus > > > > > > > > > > > > 2013/11/24 Neha Narkhede <neha.narkh...@gmail.com> > > > > > > > So, a single producer'a throughput is 80 MB/s? That seems pretty > high. > > > What > > > > was the number of acks setting? Thanks for sharing these numbers. > > > > > > > > On Sunday, November 24, 2013, Magnus Edenhill wrote: > > > > > > > > > Hi Neha, > > > > > > > > > > these tests were done using 100 byte messages. More information > about > > > the > > > > > producer performance tests can be found here: > > > > > > > > > > > > > > > > > > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers > > > > > > > > > > The tests are indicative at best and in no way scientific, but I > must > > > say > > > > > that the Kafka broker performance is impressive. > > > > > > > > > > Regards, > > > > > Magnus > > > > > > > > > > > > > > > > > > > > 2013/11/22 Neha Narkhede <neha.narkh...@gmail.com <javascript:;>> > > > > > > > > > > > Thanks for sharing this! What is the message size for the > > throughput > > > > > > numbers stated below? > > > > > > > > > > > > Thanks, > > > > > > Neha > > > > > > On Nov 22, 2013 6:59 AM, "Magnus Edenhill" <mag...@edenhill.se > > > > <javascript:;>> > > > > > wrote: > > > > > > > > > > > > > This announces the 0.8.0 release of librdkafka - The Apache > Kafka > > > > > client > > > > > > C > > > > > > > library - now with 0.8 protocol support. > > > > > > > > > > > > > > Features: > > > > > > > * Producer (~800K msgs/s) > > > > > > > * Consumer (~3M msgs/s) > > > > > > > * Compression (Snappy, gzip) > > > > > > > * Proper failover and leader re-election support - no message > is > > > ever > > > > > > lost. > > > > > > > * Configuration properties compatible with official Apache > Kafka. > > > > > > > * Stabilized ABI-safe API > > > > > > > * Mainline Debian package submitted > > > > > > > * Production quality > > > > > > > > > > > > > > > > > > > > > Home: > > > > > > > https://github.com/edenhill/librdkafka > > > > > > > > > > > > > > Introduction and performance numbers: > > > > > > > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md > > > > > > > > > > > > > > Have fun. > > > > > > > > > > > > > > Regards, > > > > > > > Magnus > > > > > > > > > > > > > > P.S. > > > > > > > Check out Wikimedia Foundation's varnishkafka daemon for a use > > > case - > > > > > > > varnish log forwarding over Kafka: > > > > > > > https://github.com/wikimedia/varnishkafka > > > > > > > > > > > > > > > > > > > > > > > > > > > >
