I don't mean to derail this at all, but given Ismael's question I'm wondering what exactly we consider a message format change? Are we assuming that additions like a new compression format (i.e. semantic changes where previously undefined bits, but bits which existed) don't require a format version change? I ask because we went the other way with protocol requests, but there are a number of outstanding KIPs (zstd compression and compaction tombstones off the top of my head) which people presumably were targeting for earlier than 0.11.0.0 but are, strictly speaking, "new" formats.
I think the different definitions might make sense, but it is probably worth clarifying. And this would have important implications. We have a bunch of proposed structural changes upcoming in the message format, but obviously we'd like to keep the number of times folks have to think about this to a minimum. On the other hand, this means that semantic changes like supporting a specific compression format might force folks to stay on an earlier broker version if they want to avoid, e.g., producers accidentally sending data in a format consumers won't understand (i.e. in this case they may want all clients upgraded before all brokers, which is where separating log format and request/response versions comes in handy). The deletion tombstones might be an even better example here since the details are probably handled automatically by the library and it only takes one upgraded client to do the wrong thing and break lots of downstream systems. In other words, we potentially tie quite a few improvements that we might otherwise think of as incremental improvements only to major version releases. This could in turn force/encourage us to do major releases more frequently and we lose the benefit of infrequent message format changes, or we still do them at the same cadence and it takes forever for new incremental features to land. -Ewen On Mon, Jan 9, 2017 at 6:34 PM, Ismael Juma <ism...@juma.me.uk> wrote: > Hi Joel, > > Great to hear that LinkedIn is likely to implement KAFKA-4513. :) > > Yes, the KIP as it stands is a compromise given the situation. We could > push the deprecation to the subsequent release: likely to be 0.11.0.0 since > there are a number of KIPs that require message format changes. This would > mean that the old consumers would not be removed before 0.12.0.0 (the major > release after 0.11.0.0). Would that work better for you all? > > Ismael > > On Tue, Jan 10, 2017 at 12:52 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > > > > > > > > The ideal scenario would be for us to provide a tool for no downtime > > > migration as discussed in the original thread (I filed > > > https://issues.apache.org/jira/browse/KAFKA-4513 in response to that > > > discussion). There are a few issues, however: > > > > > > - There doesn't seem to be much demand for it (outside of LinkedIn, > at > > > least) > > > - No-one is working on it or has indicated that they are planning to > > > work on it > > > - It's a non-trivial change and it requires a good amount of testing > > to > > > ensure it works as expected > > > > > > > For LinkedIn: while there are a few consuming applications for which the > > current shut-down and restart approach to migration will suffice, I doubt > > we will be able to do this for majority of services that are outside our > > direct control. Given that a seamless migration is a pre-req for us to > > switch to the new consumer widely (there are a few use-cases already on > it) > > it is something that we (LinkedIn) will likely implement although we > > haven't done further brainstorming/improvements beyond what was proposed > in > > the other deprecation thread. > > > > > > > In the meantime, we have this suboptimal situation where the old > > consumers > > > are close to unmaintained even though we don't say it outright: they > > don't > > > > get new features (basic things like security are missing) and bug fixes > are > > > rare. In practice, the old clients have been deprecated a while back, > we > > > > > > > Agreed that it is suboptimal, but the reality is that LI and I think a > few > > other companies are still to a large extent on the old consumer and will > be > > for at least a good part of this year. This does mean that we have the > > overhead of maintaining some internal workarounds for the old consumer. > > > > > > > just haven't made it official. This proposal is about rectifying that > so > > > that we communicate our intentions to our users more clearly. As Vahid > > > said, this KIP is not about changing how we maintain the existing code. > > > > > > The KIP that proposes the removal of all the old clients will be more > > > interesting, but it doesn't exist yet. :) > > > > > > > Deprecating *after* providing a sound migration path still seems to be > the > > right thing to do but if there isn't any demand for it then maybe that's > a > > reasonable compromise. I'm still surprised that more users aren't as > > impacted by this and as mentioned earlier, it could be an issue of > > awareness but I'm not sure that deprecating before a migration path is in > > place would be considered best-practice in raising awareness. > > > > Thanks, > > > > Joel > > > > > > > > > > > > Ismael > > > > > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian < > > > vahidhashem...@us.ibm.com > > > > wrote: > > > > > > > One thing that probably needs some clarification is what is implied > by > > > > "deprecated" in the Kafka project. > > > > I googled it a bit and it doesn't seem that deprecation > conventionally > > > > implies termination of support (or anything that could negatively > > impact > > > > existing users). That's my interpretation too. > > > > It would be good to know if Kafka follows a different interpretation > of > > > > the term. > > > > > > > > If my understanding of the term is correct, since we are not yet > > > targeting > > > > a certain major release in which the old consumer will be removed, I > > > don't > > > > see any harm in marking it as deprecated. > > > > There will be enough time to plan and implement the migration, if the > > > > community decides that's the way to go, before phasing it out. > > > > > > > > At the minimum new Kafka users will pick the Java consumer without > any > > > > confusion. And existing users will know that Kafka is preparing for > the > > > > old consumer's retirement. > > > > > > > > --Vahid > > > > > > > > > > > > > > > > > > > > From: Joel Koshy <jjkosh...@gmail.com> > > > > To: "dev@kafka.apache.org" <dev@kafka.apache.org> > > > > Date: 01/05/2017 06:55 PM > > > > Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation > > > > > > > > > > > > > > > > While I realize this only marks the old consumer as deprecated and > not > > a > > > > complete removal, I agree that it is somewhat premature to do this > > prior > > > > to > > > > having a migration process implemented. Onur has described this in > > detail > > > > in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s > > and > > > > I'm > > > > surprised that more companies aren't affected by (or aware of?) the > > > issue. > > > > > > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <radai.rosenbl...@gmail.com> > > > wrote: > > > > > > > > > I cant speak for anyone else, but a rolling upgrade is definitely > how > > > we > > > > > (LinkedIn) will do the migration. > > > > > > > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <g...@confluent.io> > > > wrote: > > > > > > > > > > > it sounds good to have > > > > > > it, but that's probably not how people will end up migrati > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
