Ok, so it looks like the default settings are: offset.storage = zookeeper dual.commit.enabled = true
The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly state) that it will only apply if offset.storage = kafka. Is that right? (I'm guessing not) "*If you are using "kafka"* as offsets.storage, you can dual commit offsets to ZooKeeper (in addition to Kafka)." It seems to me less than ideal to have the default behavior to have dual.commit.enabled = true, since this seems like a performance hit, no? I'd think you'd only want this during a planned migration. Also, I assume it's desirable to switch to using 'kafka' for offset storage, for performance reasons? Will it better handle a larger number of topics? Also, I assume the __consumer_offsets topic will be set to have an infinite retention policy internally, is that right? So that currently committed offsets for a given consumer group won't be lost? Jason On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > This is documented in the official docs: > http://kafka.apache.org/documentation.html#distributionimpl > > On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: > > What are the defaults for those settings (I assume it will be to continue > > using only zookeeper by default)? > > > > Also, if I have a cluster of consumers sharing the same groupId, and I > > update them via a rolling release, will it be a problem during the > rolling > > restart if there is inconsistency in the settings for a short time? Or > is > > it required that the entire cluster be stopped, then update configs, then > > restart all nodes? > > > > Jason > > > > On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gshap...@cloudera.com> > wrote: > > > > > Thanks Jon. I updated the FAQ with your procedure: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 > > > ? > > > > > > On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst < > > > jbringhu...@linkedin.com.invalid> wrote: > > > > > > > There should probably be a wiki page started for this so we have the > > > > details in one place. The same question was asked on Freenode IRC a > few > > > > minutes ago. :) > > > > > > > > A summary of the migration procedure is: > > > > > > > > 1) Upgrade your brokers and set dual.commit.enabled=false and > > > > offsets.storage=zookeeper (Commit offsets to Zookeeper Only). > > > > 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart > > > > (Commit offsets to Zookeeper and Kafka). > > > > 3) Set dual.commit.enabled=false and offsets.storage=kafka and > restart > > > > (Commit offsets to Kafka only). > > > > > > > > -Jon > > > > > > > > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <j...@squareup.com> > wrote: > > > > > > > > > Hi, > > > > > > > > > > For 0.8.2, one of the features listed is: > > > > > - Kafka-based offset storage. > > > > > > > > > > Is there documentation on this (I've heard discussion of it of > course)? > > > > > > > > > > Also, is it something that will be used by existing consumers when > they > > > > > migrate up to 0.8.2? What is the migration process? > > > > > > > > > > Thanks, > > > > > > > > > > Jason > > > > > > > > > > > > >
