Hey Onur, Sorry for the late reply. Thanks for the well-written KIP! I think the proposal makes sense. The only thing I was wondering is whether the process is a bit complex for most users. You'd probably have no trouble at LI (especially given you're implementing it!), but I'm not so sure about the users who aren't as close to the Kafka internals. That said, I don't see any great options to simplify the process, and having this approach is better than having none, so maybe it's fine. Here are a couple minor suggestions:
1. One thought that came to mind is whether it would be worthwhile to add a broker config to disable the group membership check for offset commits. This would simplify the process by eliminating the initial step of turning off offset commits in Kafka for the group to be migrated prior to turning on group coordination through Kafka. I'm not thrilled about this option since it removes the protection that that check provides (I guess this is no worse than using Kafka for offsets storage with the old consumer anyway). Also it's a config we'd to ultimately have to deprecate and remove. 2. It seems like the toggle on the group's coordination mode is done manually. Should we add that to consumer-groups.sh? Thanks, Jason On Thu, Feb 23, 2017 at 1:22 PM, Dong Lin <lindon...@gmail.com> wrote: > Yeah, I agree it is a bit complex to do that approach for a one-time > migration. Probably not worth it. Here is another idea to reduce, but not > eliminate, the amount of message duplication during migration. I am fine > with not doing it. Just want to see the opinion from open source community. > > The problem with current solution is that, when we toggle the zookeeper > path in order to migrate from MEZKCC, with 50% probability the old owner of > the partition may reduce notification later than the new partition owner. > Thus the new partition owner may reduce the offset committed by the older > owner 5 sec ago assuming the auto-commit interval is 10 sec. The messages > produced in this 5 sec window may be consumed multiple times. This amount > is even more if consumer is bootstrapping. > > One way to mitigate this problem is for the MEZKCC to sleep for a > configurable amount of time after it receives zookeeper notification but > before it starts to fetch offset and consume message. This seems like an > easy change that allows user to tradeoff between the message duplication > and consumer downtime. > > > > On Thu, Feb 23, 2017 at 11:20 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > Regarding (2) - yes that's a good point. @Onur - I think the KIP should > > explicitly call this out. > > It is something that we did consider and decided against optimizing for. > > i.e., we just wrote that off as a minor caveat of the upgrade path in > that > > there will be a few duplicates, but not too many given that we expect the > > period of duplicate ownership to be minimal. Although it could be > addressed > > as you described, it does add complexity to an already-rather-complex > > migration path. Given that it is a transition state (i.e., migration) we > > felt it would be better and sufficient to keep it only as complex as it > > needs to be. > > > > On Mon, Feb 20, 2017 at 4:45 PM, Onur Karaman < > > onurkaraman.apa...@gmail.com> > > wrote: > > > > > Regarding 1: We won't lose the offset from zookeeper upon partition > > > transfer from OZKCC/MDZKCC to MEZKCC because MEZKCC has > > > "dual.commit.enabled" set to true as well as "offsets.storage" set to > > > kafka. The combination of these configs results in the consumer > fetching > > > offsets from both kafka and zookeeper and just picking the greater of > the > > > two. > > > > > > On Mon, Feb 20, 2017 at 4:33 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > > > > Hey Onur, > > > > > > > > Thanks for the well-written KIP! I have two questions below. > > > > > > > > 1) In the process of migrating from OZKCCs and MDZKCCs to MEZKCCs, we > > > will > > > > may a mix of OZKCCs, MDZKCCs and MEZKCCs. OZKCC and MDZKCC will only > > > commit > > > > to zookeeper and MDZKCC will use kafka-based offset storage. Would we > > > lose > > > > offset committed to zookeeper by a MDZKCC if a partition ownership if > > > > transferred from a MDZKCC to a MEZKCC? > > > > > > > > 2) Suppose every process in the group is running MEZKCC. Each MEZKCC > > has > > > a > > > > zookeeper-based partition assignment and kafka-based partition > > > assignment. > > > > Is it guaranteed that these two assignments are exactly the same > across > > > > processes? If not, say the zookeeper-based assignment assigns p1, p2 > to > > > > process 1, and p3 to process 2. And kafka-based assignment assigns > p1, > > p3 > > > > to process 1, and p2 to process 2. Say process 1 handles receives the > > > > notification to switch to kafka-based notification before process 2, > it > > > is > > > > possible that during a short period of time p3 will be consumed by > both > > > > processes? > > > > > > > > This period is probably short and I am not sure how many messages may > > be > > > > duplicated as a result. But it seems possible to avoid this > completely > > > > according to an idea that Becket suggested in a previous discussion. > > The > > > > znode /consumers/<group id>/migration/mode can contain a sequence > > number > > > > that increment for each switch. Say the znode is toggled to kafka > with > > > > sequence number 2, each MEZKCC will commit offset to with number 2 in > > the > > > > metadata for partitions that it currently owns according to the > > zk-based > > > > partition assignment, and then periodically fetches the committed > > offset > > > > and the metadata for the partitions that it should own according to > the > > > > kafka-based partition assignment. Each MEZKCC only starts consumption > > > when > > > > the metadata has incremented to the number 2. > > > > > > > > Thanks, > > > > Dong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 20, 2017 at 12:04 PM, Onur Karaman < > > > > onurkaraman.apa...@gmail.com > > > > > wrote: > > > > > > > > > Hey everyone. > > > > > > > > > > I made a KIP that provides a mechanism for migrating from > > > > > ZookeeperConsumerConnector to KafkaConsumer as well as a mechanism > > for > > > > > rolling back from KafkaConsumer to ZookeeperConsumerConnector: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-125% > > > > > 3A+ZookeeperConsumerConnector+to+KafkaConsumer+Migration+ > > and+Rollback > > > > > > > > > > Comments are welcome. > > > > > > > > > > - Onur > > > > > > > > > > > > > > >
