Re: Post on running Kafka at LinkedIn

2015-04-08 Thread Todd Palino
...@linkedin.com.INVALID To: users@kafka.apache.org Subject: Re: Post on running Kafka at LinkedIn Date: Fri, 20 Mar 2015 22:16:29 + Keep in mind that these brokers aren't really stressed too much at any given time -- we need to stay ahead of the capacity curve. Your message throughput will really

Re: Post on running Kafka at LinkedIn

2015-03-23 Thread Todd Palino
@kafka.apache.org Subject: Re: Post on running Kafka at LinkedIn Date: Fri, 20 Mar 2015 22:16:29 + Keep in mind that these brokers aren't really stressed too much at any given time -- we need to stay ahead of the capacity curve. Your message throughput will really just depend on what hardware

Re: Post on running Kafka at LinkedIn

2015-03-20 Thread James Cheng
Amazing growth numbers. At the meetup on 1/27, Clark Haskins presented their Kafka usage at the time. It was: Bytes in: 120 TB Messages In: 585 million Bytes out: 540 TB Total brokers: 704 In Todd's post, the current numbers: Bytes in: 175 TB (45% growth) Messages In: 800 billion (36% growth)

Re: Post on running Kafka at LinkedIn

2015-03-20 Thread James Cheng
For those who missed it: The Kafka Audit tool was also presented at the 1/27 Kafka meetup: http://www.meetup.com/http-kafka-apache-org/events/219626780/ Recorded video is here, starting around the 40 minute mark: http://www.ustream.tv/recorded/58109076 Slides are here:

Re: Post on running Kafka at LinkedIn

2015-03-20 Thread Xiao
Hi, James, Thank you for sharing it! The links of videos and slides are the same. Could you check the link of slides? Xiao Li On Mar 20, 2015, at 11:30 AM, James Cheng jch...@tivo.com wrote: For those who missed it: The Kafka Audit tool was also presented at the 1/27 Kafka meetup:

Re: Post on running Kafka at LinkedIn

2015-03-20 Thread tao xiao
here is the slide http://www.slideshare.net/JonBringhurst/kafka-audit-kafka-meetup-january-27th-2015 On Sat, Mar 21, 2015 at 2:36 AM, Xiao lixiao1...@gmail.com wrote: Hi, James, Thank you for sharing it! The links of videos and slides are the same. Could you check the link of slides?

Re: Post on running Kafka at LinkedIn

2015-03-20 Thread James Cheng
Oops. Thank you Tao Xiao! -James On Mar 20, 2015, at 11:49 AM, tao xiao xiaotao...@gmail.com wrote: here is the slide http://www.slideshare.net/JonBringhurst/kafka-audit-kafka-meetup-january-27th-2015 On Sat, Mar 21, 2015 at 2:36 AM, Xiao lixiao1...@gmail.com wrote: Hi, James,

Re: Post on running Kafka at LinkedIn

2015-03-20 Thread Clark Haskins
Yep! We are growing :) -Clark Sent from my iPhone On Mar 20, 2015, at 2:14 PM, James Cheng jch...@tivo.com wrote: Amazing growth numbers. At the meetup on 1/27, Clark Haskins presented their Kafka usage at the time. It was: Bytes in: 120 TB Messages In: 585 million Bytes out: 540

RE: Post on running Kafka at LinkedIn

2015-03-20 Thread Emmanuel
800B messages / day = 9.26M messages / sec over 1100 brokers = ~8400 message / broker / sec Do I get this right? Trying to benchmark my own test cluster and that's what I see with 2 brokers...Just wondering if my numbers are good or bad... Subject: Re: Post on running Kafka at LinkedIn From

Re: Post on running Kafka at LinkedIn

2015-03-20 Thread Jon Bringhurst
/ broker / sec Do I get this right? Trying to benchmark my own test cluster and that's what I see with 2 brokers...Just wondering if my numbers are good or bad... Subject: Re: Post on running Kafka at LinkedIn From: cl...@kafka.guru Date: Fri, 20 Mar 2015 14:27:58 -0700 To: users

RE: Post on running Kafka at LinkedIn

2015-03-20 Thread Emmanuel
ridiculously low numbers (1000 events/sec/thread) So what could be causing this? From: jbringhu...@linkedin.com.INVALID To: users@kafka.apache.org Subject: Re: Post on running Kafka at LinkedIn Date: Fri, 20 Mar 2015 22:16:29 + Keep in mind that these brokers aren't really stressed too much at any