> Ok, so it looks like the default settings are: > offset.storage = zookeeper > dual.commit.enabled = true > The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly > state) that it will only apply if offset.storage = kafka. Is that right? > (I'm guessing not)
dual.commit.enabled defaults to true only if offset.storage is kafka. As you noted, it only applies if offset.storage = kafka is primarily intended for migration. > It seems to me less than ideal to have the default behavior to have > dual.commit.enabled = true, since this seems like a performance hit, no? To some degree yes, but it is relatively cheap. > I'd think you'd only want this during a planned migration. Yes. > > Also, I assume it's desirable to switch to using 'kafka' for offset > storage, for performance reasons? Will it better handle a larger number of > topics? Yes. > Also, I assume the __consumer_offsets topic will be set to have an infinite > retention policy internally, is that right? So that currently committed > offsets for a given consumer group won't be lost? It uses the "compaction" retention policy - so the topic won't grow unbounded. Compaction will basically dedupe on the inactive segments of the topic - so in effect it will only maintain the last committed offset for a given group-topic-partition 3-tuple (plus a few - since it runs only when a certain dirtiness threshold has been met). The compaction policy is also documented on the site. Thanks, Joel > On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > This is documented in the official docs: > > http://kafka.apache.org/documentation.html#distributionimpl > > > > On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: > > > What are the defaults for those settings (I assume it will be to continue > > > using only zookeeper by default)? > > > > > > Also, if I have a cluster of consumers sharing the same groupId, and I > > > update them via a rolling release, will it be a problem during the > > rolling > > > restart if there is inconsistency in the settings for a short time? Or > > is > > > it required that the entire cluster be stopped, then update configs, then > > > restart all nodes? > > > > > > Jason > > > > > > On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gshap...@cloudera.com> > > wrote: > > > > > > > Thanks Jon. I updated the FAQ with your procedure: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 > > > > ? > > > > > > > > On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst < > > > > jbringhu...@linkedin.com.invalid> wrote: > > > > > > > > > There should probably be a wiki page started for this so we have the > > > > > details in one place. The same question was asked on Freenode IRC a > > few > > > > > minutes ago. :) > > > > > > > > > > A summary of the migration procedure is: > > > > > > > > > > 1) Upgrade your brokers and set dual.commit.enabled=false and > > > > > offsets.storage=zookeeper (Commit offsets to Zookeeper Only). > > > > > 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart > > > > > (Commit offsets to Zookeeper and Kafka). > > > > > 3) Set dual.commit.enabled=false and offsets.storage=kafka and > > restart > > > > > (Commit offsets to Kafka only). > > > > > > > > > > -Jon > > > > > > > > > > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <j...@squareup.com> > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > For 0.8.2, one of the features listed is: > > > > > > - Kafka-based offset storage. > > > > > > > > > > > > Is there documentation on this (I've heard discussion of it of > > course)? > > > > > > > > > > > > Also, is it something that will be used by existing consumers when > > they > > > > > > migrate up to 0.8.2? What is the migration process? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > -- Joel
