800B messages / day = 9.26M messages / sec over 1100 brokers = ~8400 message / broker / sec Do I get this right? Trying to benchmark my own test cluster and that's what I see with 2 brokers...Just wondering if my numbers are good or bad...
> Subject: Re: Post on running Kafka at LinkedIn > From: cl...@kafka.guru > Date: Fri, 20 Mar 2015 14:27:58 -0700 > To: users@kafka.apache.org > > Yep! We are growing :) > > -Clark > > Sent from my iPhone > > > On Mar 20, 2015, at 2:14 PM, James Cheng <jch...@tivo.com> wrote: > > > > Amazing growth numbers. > > > > At the meetup on 1/27, Clark Haskins presented their Kafka usage at the > > time. It was: > > > > Bytes in: 120 TB > > Messages In: 585 million > > Bytes out: 540 TB > > Total brokers: 704 > > > > In Todd's post, the current numbers: > > > > Bytes in: 175 TB (45% growth) > > Messages In: 800 billion (36% growth) > > Bytes out: 650 TB (20% growth) > > Total brokers: 1100 (56% growth) > > > > That much growth in just 2 months? Wowzers. > > > > -James > > > >> On Mar 20, 2015, at 11:30 AM, James Cheng <jch...@tivo.com> wrote: > >> > >> For those who missed it: > >> > >> The Kafka Audit tool was also presented at the 1/27 Kafka meetup: > >> http://www.meetup.com/http-kafka-apache-org/events/219626780/ > >> > >> Recorded video is here, starting around the 40 minute mark: > >> http://www.ustream.tv/recorded/58109076 > >> > >> Slides are here: > >> http://www.ustream.tv/recorded/58109076 > >> > >> -James > >> > >>> On Mar 20, 2015, at 9:47 AM, Todd Palino <tpal...@gmail.com> wrote: > >>> > >>> For those who are interested in detail on how we've got Kafka set up at > >>> LinkedIn, I have just published a new posted to our Engineering blog > >>> titled > >>> "Running Kafka at Scale" > >>> > >>> https://engineering.linkedin.com/kafka/running-kafka-scale > >>> > >>> It's a general overview of our current Kafka install, tiered architecture, > >>> audit, and the libraries we use for producers and consumers. You'll also > >>> be > >>> seeing more posts from the SRE team here in the coming weeks on deeper > >>> looks into both Kafka and Samza. > >>> > >>> Additionally, I'll be giving a talk at ApacheCon next month on running > >>> tiered Kafka architectures. If you're in Austin for that, please come by > >>> and check it out. > >>> > >>> -Todd > >
