Becket, Please let us know once you have updated the wiki page regarding the migration plan. Thanks!
Guozhang On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira <g...@confluent.io> wrote: > Thanks Becket, Anne and Neha for responding to my concern. > > I had an offline discussion with Anne where she helped me understand the > migration process. It isn't as bad as it looks in the KIP :) > > If I understand it correctly, the process (for users) will be: > > 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) > 2. Rolling upgrade of brokers > 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers will use > the new protocol > 4. Start upgrading clients > 5. When "enough" clients are upgraded, bump format.version to 1 (rolling). > > Becket, can you confirm? > > Assuming this is the process, I'm +1 on the change. > > Reminder to coders and reviewers that pull-requests with user-facing > changes should include documentation changes as well as code changes. > And a polite request to try to be helpful to users on when to use > create-time and when to use log-append-time as configuration - this is not > a trivial decision. > > A separate point I'm going to raise in a different thread is that we need > to streamline our versions a bit: > 1. I'm afraid that 0.9.0-1 will be confusing to users who care about > released versions (what if we forget to change it before the release? Is it > meaningful enough to someone running off trunk?), we need to come up with > something that will work for both LinkedIn and everyone else. > 2. ApiVersion has real version numbers. message.format.version has sequence > numbers. This makes us look pretty silly :) > > My version concerns can be addressed separately and should not hold back > this KIP. > > Gwen > > > > On Tue, Dec 22, 2015 at 11:01 PM, Becket Qin <becket....@gmail.com> wrote: > > > Hi Anna, > > > > Thanks for initiating the voting process. I did not start the voting > > process because there were still some ongoing discussion with Jun about > the > > timestamp regarding compressed messages. That is why the wiki page hasn't > > reflected the latest conversation as Guozhang pointed out. > > > > Like Neha said I think we have reached general agreement on this KIP. So > > it is probably fine to start the KIP voting. At least we draw more > > attention to the KIP even if there are some new discussion to bring up. > > > > Regarding the upgrade plan, given we decided to implement KIP-31 and > > KIP-32 in the same patch to avoid change binary protocol twice, the > upgrade > > plan was mostly discussed on the discussion thread of KIP-31. > > > > Since the voting has been initiated, I will update the wiki with latest > > conversation to avoid further confusion. > > > > BTW, I actually have started coding work on KIP-31 and KIP-32 and will > > focus on the patch before I return from vacation in mid Jan because I > have > > no LInkedIn VPN access in China anyway... > > > > Thanks, > > > > Jiangjie > > > > > On Dec 23, 2015, at 12:31 PM, Anna Povzner <a...@confluent.io> wrote: > > > > > > Hi Gwen, > > > > > > I just wanted to point out that I just started the vote. Becket wrote > the > > > proposal and led the discussions. > > > > > > What I understood from reading the discussion thread, the migration > plan > > > was discussed at the KIP meeting, and not much on the mailing list > > itself. > > > > > > My question about the migration plan was same as Guozhang wrote: The > case > > > when an upgraded broker receives an old producer request. The proposal > is > > > for the broker to fill in the timestamp field with the current time at > > the > > > broker. Cons: it goes against the definition of CreateTime type of the > > > timestamp (we are "over-writing" it at the broker). Pros: It looks like > > > most of the use-cases would actually want that behavior, because > > otherwise > > > timestamp is useless and they will need to support an old, > pre-timestamp, > > > behavior. E.g., if we modify log retention policy to use the timestamp, > > we > > > would need to support an old implementation (the one that does not use > > > timestamps in the message). So I actually have a preference for the > > > proposed approach. > > > > > > Thanks, > > > Anna > > > > > >> On Tue, Dec 22, 2015 at 8:02 PM, Neha Narkhede <n...@confluent.io> > > wrote: > > >> > > >> Hey Gwen, > > >> > > >> Migration plan wasn't really discussed a ton in the previous threads. > > So it > > >> will be great to dive deep and see if there are gaps there. I had some > > >> questions, but the details listed on the KIP are great. > > >> > > >> It is complex, though the plan outlined in the wiki assumes a zero > > downtime > > >> upgrade assuming that producers and consumers can't be upgraded in > > tandem. > > >> This is typical for companies that have a significant Kafka footprint, > > like > > >> LinkedIn. > > >> > > >> Thanks, > > >> Neha > > >> > > >>> On Tue, Dec 22, 2015 at 7:48 PM, Gwen Shapira <g...@confluent.io> > > wrote: > > >>> > > >>> Hi Anna, > > >>> > > >>> Thanks for the KIP, especially for the details on all the > alternatives > > >> and > > >>> how we arrived at the proposal. Its really great! > > >>> > > >>> Can you point me at where the migration plan was discussed? It looks > > >> overly > > >>> complex and I have a bunch of questions, but if there was a > discussion, > > >> I'd > > >>> like to read up rather than repeating it :) > > >>> > > >>> Gwen > > >>> > > >>>> On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner <a...@confluent.io> > > >>> wrote: > > >>> > > >>>> Hi, > > >>>> > > >>>> I am opening the voting thread for KIP-32: Add CreateTime and > > >>>> LogAppendTime to Kafka message. > > >>>> > > >>>> For reference, here's the KIP wiki: > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message > > >>>> > > >>>> And the mailing list threads: > > >>>> > > >>>> September: > > >> > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E > > >>>> > > >>>> October: > > >> > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E > > >>>> > > >>>> December: > > >> > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E > > >>>> > > >>>> > > >>>> Thanks > > >>>> Anna > > >> > > >> > > >> > > >> -- > > >> Thanks, > > >> Neha > > >> > > > -- -- Guozhang
