On Fri, Feb 06, 2015 at 12:43:37AM -0500, Jason Rosenberg wrote: > I'm not sure what you mean by 'default' behavior 'only if' offset.storage > is kafka. Does that mean the 'default' behavior is 'false' if > offset.storage is 'zookeeper'? Can that be clarified in the config > documentation section? > > In section 5.6 where the offset managements is described, there is this: > "A roll-back (i.e., migrating from Kafka back to ZooKeeper) can also be > performed using the above steps if you set offsets.storage=zookeeper." > > This implies that dual commit will work also if offsets.storage=zookeeper, > no? Just not by default? Perhaps there needs to be clarification there > (and in the config section for offsets.storage & dual.commit.enabled).
Actually I think there may be a bug here if someone needs to roll back from Kafka-based offsets to zookeeper. Will reply tomorrow on this. > > The doc in section 5.6 is probably in need of editing, it looks like it in > places assumes zookeeper offset storage, and has some repeated sentences, > etc. Will do. > > Finally, why is section 5.6 titled "Distribution"? Seems to be a grab-bag > of mostly consumer related topics? Yes this was prior structure that can be improved. > > > > > On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > > > > > This is documented in the official docs: > > > > http://kafka.apache.org/documentation.html#distributionimpl > > > > > > > > On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: > > > > > What are the defaults for those settings (I assume it will be to > > continue > > > > > using only zookeeper by default)? > > > > > > > > > > Also, if I have a cluster of consumers sharing the same groupId, and > > I > > > > > update them via a rolling release, will it be a problem during the > > > > rolling > > > > > restart if there is inconsistency in the settings for a short time? > > Or > > > > is > > > > > it required that the entire cluster be stopped, then update configs, > > then > > > > > restart all nodes? > > > > > > > > > > Jason > > > > > > > > > > On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gshap...@cloudera.com > > > > > > > wrote: > > > > > > > > > > > Thanks Jon. I updated the FAQ with your procedure: > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 > > > > > > ? > > > > > > > > > > > > On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst < > > > > > > jbringhu...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > There should probably be a wiki page started for this so we have > > the > > > > > > > details in one place. The same question was asked on Freenode > > IRC a > > > > few > > > > > > > minutes ago. :) > > > > > > > > > > > > > > A summary of the migration procedure is: > > > > > > > > > > > > > > 1) Upgrade your brokers and set dual.commit.enabled=false and > > > > > > > offsets.storage=zookeeper (Commit offsets to Zookeeper Only). > > > > > > > 2) Set dual.commit.enabled=true and offsets.storage=kafka and > > restart > > > > > > > (Commit offsets to Zookeeper and Kafka). > > > > > > > 3) Set dual.commit.enabled=false and offsets.storage=kafka and > > > > restart > > > > > > > (Commit offsets to Kafka only). > > > > > > > > > > > > > > -Jon > > > > > > > > > > > > > > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <j...@squareup.com> > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > For 0.8.2, one of the features listed is: > > > > > > > > - Kafka-based offset storage. > > > > > > > > > > > > > > > > Is there documentation on this (I've heard discussion of it of > > > > course)? > > > > > > > > > > > > > > > > Also, is it something that will be used by existing consumers > > when > > > > they > > > > > > > > migrate up to 0.8.2? What is the migration process? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Joel > >
