Re: question about new consumer offset management in 0.8.2
On Fri, Feb 06, 2015 at 12:43:37AM -0500, Jason Rosenberg wrote: I'm not sure what you mean by 'default' behavior 'only if' offset.storage is kafka. Does that mean the 'default' behavior is 'false' if offset.storage is 'zookeeper'? Can that be clarified in the config documentation section? In section 5.6 where the offset managements is described, there is this: A roll-back (i.e., migrating from Kafka back to ZooKeeper) can also be performed using the above steps if you set offsets.storage=zookeeper. This implies that dual commit will work also if offsets.storage=zookeeper, no? Just not by default? Perhaps there needs to be clarification there (and in the config section for offsets.storage dual.commit.enabled). Actually I think there may be a bug here if someone needs to roll back from Kafka-based offsets to zookeeper. Will reply tomorrow on this. The doc in section 5.6 is probably in need of editing, it looks like it in places assumes zookeeper offset storage, and has some repeated sentences, etc. Will do. Finally, why is section 5.6 titled Distribution? Seems to be a grab-bag of mostly consumer related topics? Yes this was prior structure that can be improved. On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy jjkosh...@gmail.com wrote: This is documented in the official docs: http://kafka.apache.org/documentation.html#distributionimpl On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: What are the defaults for those settings (I assume it will be to continue using only zookeeper by default)? Also, if I have a cluster of consumers sharing the same groupId, and I update them via a rolling release, will it be a problem during the rolling restart if there is inconsistency in the settings for a short time? Or is it required that the entire cluster be stopped, then update configs, then restart all nodes? Jason On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira gshap...@cloudera.com wrote: Thanks Jon. I updated the FAQ with your procedure: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 ? On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst jbringhu...@linkedin.com.invalid wrote: There should probably be a wiki page started for this so we have the details in one place. The same question was asked on Freenode IRC a few minutes ago. :) A summary of the migration procedure is: 1) Upgrade your brokers and set dual.commit.enabled=false and offsets.storage=zookeeper (Commit offsets to Zookeeper Only). 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart (Commit offsets to Zookeeper and Kafka). 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart (Commit offsets to Kafka only). -Jon On Feb 5, 2015, at 9:03 AM, Jason Rosenberg j...@squareup.com wrote: Hi, For 0.8.2, one of the features listed is: - Kafka-based offset storage. Is there documentation on this (I've heard discussion of it of course)? Also, is it something that will be used by existing consumers when they migrate up to 0.8.2? What is the migration process? Thanks, Jason -- Joel
Re: question about new consumer offset management in 0.8.2
On Thu, Feb 05, 2015 at 11:57:15PM -0800, Joel Koshy wrote: On Fri, Feb 06, 2015 at 12:43:37AM -0500, Jason Rosenberg wrote: I'm not sure what you mean by 'default' behavior 'only if' offset.storage is kafka. Does that mean the 'default' behavior is 'false' if offset.storage is 'zookeeper'? Can that be clarified in the config documentation section? In section 5.6 where the offset managements is described, there is this: A roll-back (i.e., migrating from Kafka back to ZooKeeper) can also be performed using the above steps if you set offsets.storage=zookeeper. This implies that dual commit will work also if offsets.storage=zookeeper, no? Just not by default? Perhaps there needs to be clarification there (and in the config section for offsets.storage dual.commit.enabled). Actually I think there may be a bug here if someone needs to roll back from Kafka-based offsets to zookeeper. Will reply tomorrow on this. Never mind - I think we are fine here. The scenario I was thinking about is the following: - If there are three consumer instances c0, c1, c2 consuming partitions pX, pY, ... and are committing offsets to Kafka and you want to migrate to zookeeper - Do a rolling bounce to turn on dual-commit (and keep offset.storage = kafka) - Do another rolling bounce to set offset.storage to zookeeper: - Say, you bounce c0 to commit offsets to zk and it comes back up and then owns pX. It begins to commit offsets for pX to zookeeper only. - You then bounce c1; after it goes down due to our partition assignment strategy say pX is now assigned to c2 (which has not yet been bounced). - c2 uses offset.storage=kafka so would fetch a potentially stale offset for pX which would be an issue. - So we explicitly handle this case - if dual.commit is turned on and offset.storage is kafka, then the broker fetches offsets from both Kafka and ZooKeeper and selects the maximum of the two. Let me know if you see any holes in the above. dual.commit is confusing and would have been (slightly) less confusing if it was called offset.migration.in.progress or something similar. Still, I think we can document the process carefully and state clearly that it is intended for use during migration/roll-back only.
Re: question about new consumer offset management in 0.8.2
On Thu, Feb 5, 2015 at 9:52 PM, Joel Koshy jjkosh...@gmail.com wrote: Ok, so it looks like the default settings are: offset.storage = zookeeper dual.commit.enabled = true The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly state) that it will only apply if offset.storage = kafka. Is that right? (I'm guessing not) dual.commit.enabled defaults to true only if offset.storage is kafka. As you noted, it only applies if offset.storage = kafka is primarily intended for migration. I'm not sure what you mean by 'default' behavior 'only if' offset.storage is kafka. Does that mean the 'default' behavior is 'false' if offset.storage is 'zookeeper'? Can that be clarified in the config documentation section? In section 5.6 where the offset managements is described, there is this: A roll-back (i.e., migrating from Kafka back to ZooKeeper) can also be performed using the above steps if you set offsets.storage=zookeeper. This implies that dual commit will work also if offsets.storage=zookeeper, no? Just not by default? Perhaps there needs to be clarification there (and in the config section for offsets.storage dual.commit.enabled). The doc in section 5.6 is probably in need of editing, it looks like it in places assumes zookeeper offset storage, and has some repeated sentences, etc. Finally, why is section 5.6 titled Distribution? Seems to be a grab-bag of mostly consumer related topics? It seems to me less than ideal to have the default behavior to have dual.commit.enabled = true, since this seems like a performance hit, no? To some degree yes, but it is relatively cheap. And as you pointed out (I think) it's not even an issue because it won't be dual committing initially, by default (while offsets.storage = zookeeper), right? Also, I assume the __consumer_offsets topic will be set to have an infinite retention policy internally, is that right? So that currently committed offsets for a given consumer group won't be lost? It uses the compaction retention policy - so the topic won't grow unbounded. Nice Thanks, Joel On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy jjkosh...@gmail.com wrote: This is documented in the official docs: http://kafka.apache.org/documentation.html#distributionimpl On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: What are the defaults for those settings (I assume it will be to continue using only zookeeper by default)? Also, if I have a cluster of consumers sharing the same groupId, and I update them via a rolling release, will it be a problem during the rolling restart if there is inconsistency in the settings for a short time? Or is it required that the entire cluster be stopped, then update configs, then restart all nodes? Jason On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira gshap...@cloudera.com wrote: Thanks Jon. I updated the FAQ with your procedure: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 ? On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst jbringhu...@linkedin.com.invalid wrote: There should probably be a wiki page started for this so we have the details in one place. The same question was asked on Freenode IRC a few minutes ago. :) A summary of the migration procedure is: 1) Upgrade your brokers and set dual.commit.enabled=false and offsets.storage=zookeeper (Commit offsets to Zookeeper Only). 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart (Commit offsets to Zookeeper and Kafka). 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart (Commit offsets to Kafka only). -Jon On Feb 5, 2015, at 9:03 AM, Jason Rosenberg j...@squareup.com wrote: Hi, For 0.8.2, one of the features listed is: - Kafka-based offset storage. Is there documentation on this (I've heard discussion of it of course)? Also, is it something that will be used by existing consumers when they migrate up to 0.8.2? What is the migration process? Thanks, Jason -- Joel
Re: question about new consumer offset management in 0.8.2
Ok, so it looks like the default settings are: offset.storage = zookeeper dual.commit.enabled = true The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly state) that it will only apply if offset.storage = kafka. Is that right? (I'm guessing not) *If you are using kafka* as offsets.storage, you can dual commit offsets to ZooKeeper (in addition to Kafka). It seems to me less than ideal to have the default behavior to have dual.commit.enabled = true, since this seems like a performance hit, no? I'd think you'd only want this during a planned migration. Also, I assume it's desirable to switch to using 'kafka' for offset storage, for performance reasons? Will it better handle a larger number of topics? Also, I assume the __consumer_offsets topic will be set to have an infinite retention policy internally, is that right? So that currently committed offsets for a given consumer group won't be lost? Jason On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy jjkosh...@gmail.com wrote: This is documented in the official docs: http://kafka.apache.org/documentation.html#distributionimpl On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: What are the defaults for those settings (I assume it will be to continue using only zookeeper by default)? Also, if I have a cluster of consumers sharing the same groupId, and I update them via a rolling release, will it be a problem during the rolling restart if there is inconsistency in the settings for a short time? Or is it required that the entire cluster be stopped, then update configs, then restart all nodes? Jason On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira gshap...@cloudera.com wrote: Thanks Jon. I updated the FAQ with your procedure: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 ? On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst jbringhu...@linkedin.com.invalid wrote: There should probably be a wiki page started for this so we have the details in one place. The same question was asked on Freenode IRC a few minutes ago. :) A summary of the migration procedure is: 1) Upgrade your brokers and set dual.commit.enabled=false and offsets.storage=zookeeper (Commit offsets to Zookeeper Only). 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart (Commit offsets to Zookeeper and Kafka). 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart (Commit offsets to Kafka only). -Jon On Feb 5, 2015, at 9:03 AM, Jason Rosenberg j...@squareup.com wrote: Hi, For 0.8.2, one of the features listed is: - Kafka-based offset storage. Is there documentation on this (I've heard discussion of it of course)? Also, is it something that will be used by existing consumers when they migrate up to 0.8.2? What is the migration process? Thanks, Jason
Re: question about new consumer offset management in 0.8.2
This is documented in the official docs: http://kafka.apache.org/documentation.html#distributionimpl On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: What are the defaults for those settings (I assume it will be to continue using only zookeeper by default)? Also, if I have a cluster of consumers sharing the same groupId, and I update them via a rolling release, will it be a problem during the rolling restart if there is inconsistency in the settings for a short time? Or is it required that the entire cluster be stopped, then update configs, then restart all nodes? Jason On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira gshap...@cloudera.com wrote: Thanks Jon. I updated the FAQ with your procedure: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 ? On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst jbringhu...@linkedin.com.invalid wrote: There should probably be a wiki page started for this so we have the details in one place. The same question was asked on Freenode IRC a few minutes ago. :) A summary of the migration procedure is: 1) Upgrade your brokers and set dual.commit.enabled=false and offsets.storage=zookeeper (Commit offsets to Zookeeper Only). 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart (Commit offsets to Zookeeper and Kafka). 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart (Commit offsets to Kafka only). -Jon On Feb 5, 2015, at 9:03 AM, Jason Rosenberg j...@squareup.com wrote: Hi, For 0.8.2, one of the features listed is: - Kafka-based offset storage. Is there documentation on this (I've heard discussion of it of course)? Also, is it something that will be used by existing consumers when they migrate up to 0.8.2? What is the migration process? Thanks, Jason
Re: question about new consumer offset management in 0.8.2
This is what I've found so far. https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka The high-level consumer just worked for me by setting offsets.storage = kafka. Scroll down to the offsets.* config params. http://kafka.apache.org/documentation.html#consumerconfigs -Suren On Thu, Feb 5, 2015 at 12:16 PM, Jon Bringhurst jbringhu...@linkedin.com.invalid wrote: There should probably be a wiki page started for this so we have the details in one place. The same question was asked on Freenode IRC a few minutes ago. :) A summary of the migration procedure is: 1) Upgrade your brokers and set dual.commit.enabled=false and offsets.storage=zookeeper (Commit offsets to Zookeeper Only). 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart (Commit offsets to Zookeeper and Kafka). 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart (Commit offsets to Kafka only). -Jon On Feb 5, 2015, at 9:03 AM, Jason Rosenberg j...@squareup.com wrote: Hi, For 0.8.2, one of the features listed is: - Kafka-based offset storage. Is there documentation on this (I've heard discussion of it of course)? Also, is it something that will be used by existing consumers when they migrate up to 0.8.2? What is the migration process? Thanks, Jason -- SUREN HIRAMAN, VP TECHNOLOGY Velos Accelerating Machine Learning 54 West 40th Street, 3RD FLOOR NEW YORK, NY 10018 T: @suren_h E: suren.hiraman@v suren.hira...@sociocast.comelos.io W: www.velos.io
Re: question about new consumer offset management in 0.8.2
There should probably be a wiki page started for this so we have the details in one place. The same question was asked on Freenode IRC a few minutes ago. :) A summary of the migration procedure is: 1) Upgrade your brokers and set dual.commit.enabled=false and offsets.storage=zookeeper (Commit offsets to Zookeeper Only). 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart (Commit offsets to Zookeeper and Kafka). 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart (Commit offsets to Kafka only). -Jon On Feb 5, 2015, at 9:03 AM, Jason Rosenberg j...@squareup.com wrote: Hi, For 0.8.2, one of the features listed is: - Kafka-based offset storage. Is there documentation on this (I've heard discussion of it of course)? Also, is it something that will be used by existing consumers when they migrate up to 0.8.2? What is the migration process? Thanks, Jason signature.asc Description: Message signed with OpenPGP using GPGMail
Re: question about new consumer offset management in 0.8.2
Ok, so it looks like the default settings are: offset.storage = zookeeper dual.commit.enabled = true The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly state) that it will only apply if offset.storage = kafka. Is that right? (I'm guessing not) dual.commit.enabled defaults to true only if offset.storage is kafka. As you noted, it only applies if offset.storage = kafka is primarily intended for migration. It seems to me less than ideal to have the default behavior to have dual.commit.enabled = true, since this seems like a performance hit, no? To some degree yes, but it is relatively cheap. I'd think you'd only want this during a planned migration. Yes. Also, I assume it's desirable to switch to using 'kafka' for offset storage, for performance reasons? Will it better handle a larger number of topics? Yes. Also, I assume the __consumer_offsets topic will be set to have an infinite retention policy internally, is that right? So that currently committed offsets for a given consumer group won't be lost? It uses the compaction retention policy - so the topic won't grow unbounded. Compaction will basically dedupe on the inactive segments of the topic - so in effect it will only maintain the last committed offset for a given group-topic-partition 3-tuple (plus a few - since it runs only when a certain dirtiness threshold has been met). The compaction policy is also documented on the site. Thanks, Joel On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy jjkosh...@gmail.com wrote: This is documented in the official docs: http://kafka.apache.org/documentation.html#distributionimpl On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: What are the defaults for those settings (I assume it will be to continue using only zookeeper by default)? Also, if I have a cluster of consumers sharing the same groupId, and I update them via a rolling release, will it be a problem during the rolling restart if there is inconsistency in the settings for a short time? Or is it required that the entire cluster be stopped, then update configs, then restart all nodes? Jason On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira gshap...@cloudera.com wrote: Thanks Jon. I updated the FAQ with your procedure: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 ? On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst jbringhu...@linkedin.com.invalid wrote: There should probably be a wiki page started for this so we have the details in one place. The same question was asked on Freenode IRC a few minutes ago. :) A summary of the migration procedure is: 1) Upgrade your brokers and set dual.commit.enabled=false and offsets.storage=zookeeper (Commit offsets to Zookeeper Only). 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart (Commit offsets to Zookeeper and Kafka). 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart (Commit offsets to Kafka only). -Jon On Feb 5, 2015, at 9:03 AM, Jason Rosenberg j...@squareup.com wrote: Hi, For 0.8.2, one of the features listed is: - Kafka-based offset storage. Is there documentation on this (I've heard discussion of it of course)? Also, is it something that will be used by existing consumers when they migrate up to 0.8.2? What is the migration process? Thanks, Jason -- Joel
Re: question about new consumer offset management in 0.8.2
Thanks Jon. I updated the FAQ with your procedure: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 ? On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst jbringhu...@linkedin.com.invalid wrote: There should probably be a wiki page started for this so we have the details in one place. The same question was asked on Freenode IRC a few minutes ago. :) A summary of the migration procedure is: 1) Upgrade your brokers and set dual.commit.enabled=false and offsets.storage=zookeeper (Commit offsets to Zookeeper Only). 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart (Commit offsets to Zookeeper and Kafka). 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart (Commit offsets to Kafka only). -Jon On Feb 5, 2015, at 9:03 AM, Jason Rosenberg j...@squareup.com wrote: Hi, For 0.8.2, one of the features listed is: - Kafka-based offset storage. Is there documentation on this (I've heard discussion of it of course)? Also, is it something that will be used by existing consumers when they migrate up to 0.8.2? What is the migration process? Thanks, Jason
Re: question about new consumer offset management in 0.8.2
What are the defaults for those settings (I assume it will be to continue using only zookeeper by default)? Also, if I have a cluster of consumers sharing the same groupId, and I update them via a rolling release, will it be a problem during the rolling restart if there is inconsistency in the settings for a short time? Or is it required that the entire cluster be stopped, then update configs, then restart all nodes? Jason On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira gshap...@cloudera.com wrote: Thanks Jon. I updated the FAQ with your procedure: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2 ? On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst jbringhu...@linkedin.com.invalid wrote: There should probably be a wiki page started for this so we have the details in one place. The same question was asked on Freenode IRC a few minutes ago. :) A summary of the migration procedure is: 1) Upgrade your brokers and set dual.commit.enabled=false and offsets.storage=zookeeper (Commit offsets to Zookeeper Only). 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart (Commit offsets to Zookeeper and Kafka). 3) Set dual.commit.enabled=false and offsets.storage=kafka and restart (Commit offsets to Kafka only). -Jon On Feb 5, 2015, at 9:03 AM, Jason Rosenberg j...@squareup.com wrote: Hi, For 0.8.2, one of the features listed is: - Kafka-based offset storage. Is there documentation on this (I've heard discussion of it of course)? Also, is it something that will be used by existing consumers when they migrate up to 0.8.2? What is the migration process? Thanks, Jason
