At LinkedIn, we are just beginning to move our applications to Kafka based offset management. I'm not sure if Todd/Clark/Joel have any numbers to share right away, but depending on the number of consumers you have and the number of topics those consumers pull, it will likely make a sizable impact on the performance of your zookeeper cluster. It will not, however, influence the decision of sizing the zookeeper cluster since zookeeper writes do not scale with the number of servers in a zookeeper cluster. In general, it is recommended to keep a zookeeper cluster small, of the size of 3 or 5 servers. i.e. whether or not your consumers checkpoint offsets to zookeeper or Kafka. It is true that the writes will be faster on a 3 node zookeeper cluster vs a 5 node zookeeper cluster.
Thanks Neha On Wed, Jun 4, 2014 at 11:15 AM, S Ahmed <sahmed1...@gmail.com> wrote: > Very nice. > > Do you guys have any stats on what kind of load was reduced on ZK? Just > trying to understand if this changes the type of servers required to host > ZK. > > > > > On Wed, Jun 4, 2014 at 1:10 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Yes, we are migrating the offset management from ZK to the broker as a > > special log. > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management > > > > The code is in trunk, and it is running in production at LinkedIn now. > > > > Guozhang > > > > > > On Wed, Jun 4, 2014 at 10:00 AM, S Ahmed <sahmed1...@gmail.com> wrote: > > > > > I swear I read that Jay Kreps wrote somewhere that consumers now write > > > their offsets in a logfile (not in zookeeper). > > > > > > Is this true or did I misread? Sorry I can't find the article I was > > > reading. > > > > > > > > > > > -- > > -- Guozhang > > >
