It might also be helpful for anyone who has done a bit of thinking around the migration story to dump their thoughts into the JIRA re: the plan. There are also questions I would have around what the exact requirements are. For example, if supporting auto commit is required, things get a lot hairier. If commits are explicit and standard partition assignors are used, then we may be able to get away with something relatively cheap (implementation-wise), e.g. you deploy with both clients but can start accepting data from the new consumer as soon as assignments match between the old & new (with some coordination to deal with timing differences between groups).
It's really hard to judge how aggressive we might want to be with deprecating without any idea what the LOE is to provide the transition tool, and difficult to plan for building the transition tool as a prereq for deprecation if we don't have any rough LOE. If the LOE is small enough, maybe deprecating now is fine as long as we still provide a big enough window for transition after the tool is built. I also think it's worth pointing out that the other important impact of deprecation is that folks should stop writing *new* code with the deprecated consumer. If we can encourage folks to start using the new consumer now, it makes their transition experience better since they have fewer apps to move. For this reason alone, I'd like to deprecate the old consumer before having the full migration story (with the docs/release note that we'll provide something with plenty of time to make the transition before removing the client entirely). -Ewen On Tue, Jan 10, 2017 at 2:11 PM, Ismael Juma <ism...@juma.me.uk> wrote: > Hi Renu, > > 0.10 was released in May. 2016. 0.11 will be not be before May 2017 (it > could be later if the next release turns out to be 0.10.3). So the most > recent data indicates a minimum of 1 year between major releases, but no > decision has been made on future major release yet. > > The impact would indeed be build warnings, but those can be suppressed > easily in the build config or via @SupressWarnings annotations. Do you use > the consumer API directly or do you have a company wrapper? > > Ismael > > On Tue, Jan 10, 2017 at 8:53 PM, Renu Tewari <tewa...@gmail.com> wrote: > > > Hi Ismael, > > What are the expected timelines we are talking about between the major > > releases? At LI we are expecting to have atleast 1 year between the old > > consumer deprecation and removal so we have enough time to upgrade all > > applications. The rollout to new consumer has hit many hurdles so hasn't > > proceeded at the expected pace. What impact would an official deprecation > > have on applications? Any warnings would be disruptive and would prefer > > that happens when there is a migration plan in place so we have a bound > on > > how long it will take. There are too many unknowns at this time. > > > > Thoughts on timelines? > > > > regards > > Renu > > > > On Mon, Jan 9, 2017 at 6:34 PM, Ismael Juma <ism...@juma.me.uk> wrote: > > > > > Hi Joel, > > > > > > Great to hear that LinkedIn is likely to implement KAFKA-4513. :) > > > > > > Yes, the KIP as it stands is a compromise given the situation. We could > > > push the deprecation to the subsequent release: likely to be 0.11.0.0 > > since > > > there are a number of KIPs that require message format changes. This > > would > > > mean that the old consumers would not be removed before 0.12.0.0 (the > > major > > > release after 0.11.0.0). Would that work better for you all? > > > > > > Ismael > > > > > > On Tue, Jan 10, 2017 at 12:52 AM, Joel Koshy <jjkosh...@gmail.com> > > wrote: > > > > > > > > > > > > > > > > > > The ideal scenario would be for us to provide a tool for no > downtime > > > > > migration as discussed in the original thread (I filed > > > > > https://issues.apache.org/jira/browse/KAFKA-4513 in response to > that > > > > > discussion). There are a few issues, however: > > > > > > > > > > - There doesn't seem to be much demand for it (outside of > > LinkedIn, > > > at > > > > > least) > > > > > - No-one is working on it or has indicated that they are > planning > > to > > > > > work on it > > > > > - It's a non-trivial change and it requires a good amount of > > testing > > > > to > > > > > ensure it works as expected > > > > > > > > > > > > > For LinkedIn: while there are a few consuming applications for which > > the > > > > current shut-down and restart approach to migration will suffice, I > > doubt > > > > we will be able to do this for majority of services that are outside > > our > > > > direct control. Given that a seamless migration is a pre-req for us > to > > > > switch to the new consumer widely (there are a few use-cases already > on > > > it) > > > > it is something that we (LinkedIn) will likely implement although we > > > > haven't done further brainstorming/improvements beyond what was > > proposed > > > in > > > > the other deprecation thread. > > > > > > > > > > > > > In the meantime, we have this suboptimal situation where the old > > > > consumers > > > > > are close to unmaintained even though we don't say it outright: > they > > > > don't > > > > > > > > get new features (basic things like security are missing) and bug > fixes > > > are > > > > > rare. In practice, the old clients have been deprecated a while > back, > > > we > > > > > > > > > > > > > Agreed that it is suboptimal, but the reality is that LI and I think > a > > > few > > > > other companies are still to a large extent on the old consumer and > > will > > > be > > > > for at least a good part of this year. This does mean that we have > the > > > > overhead of maintaining some internal workarounds for the old > consumer. > > > > > > > > > > > > > just haven't made it official. This proposal is about rectifying > that > > > so > > > > > that we communicate our intentions to our users more clearly. As > > Vahid > > > > > said, this KIP is not about changing how we maintain the existing > > code. > > > > > > > > > > The KIP that proposes the removal of all the old clients will be > more > > > > > interesting, but it doesn't exist yet. :) > > > > > > > > > > > > > Deprecating *after* providing a sound migration path still seems to > be > > > the > > > > right thing to do but if there isn't any demand for it then maybe > > that's > > > a > > > > reasonable compromise. I'm still surprised that more users aren't as > > > > impacted by this and as mentioned earlier, it could be an issue of > > > > awareness but I'm not sure that deprecating before a migration path > is > > in > > > > place would be considered best-practice in raising awareness. > > > > > > > > Thanks, > > > > > > > > Joel > > > > > > > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian < > > > > > vahidhashem...@us.ibm.com > > > > > > wrote: > > > > > > > > > > > One thing that probably needs some clarification is what is > implied > > > by > > > > > > "deprecated" in the Kafka project. > > > > > > I googled it a bit and it doesn't seem that deprecation > > > conventionally > > > > > > implies termination of support (or anything that could negatively > > > > impact > > > > > > existing users). That's my interpretation too. > > > > > > It would be good to know if Kafka follows a different > > interpretation > > > of > > > > > > the term. > > > > > > > > > > > > If my understanding of the term is correct, since we are not yet > > > > > targeting > > > > > > a certain major release in which the old consumer will be > removed, > > I > > > > > don't > > > > > > see any harm in marking it as deprecated. > > > > > > There will be enough time to plan and implement the migration, if > > the > > > > > > community decides that's the way to go, before phasing it out. > > > > > > > > > > > > At the minimum new Kafka users will pick the Java consumer > without > > > any > > > > > > confusion. And existing users will know that Kafka is preparing > for > > > the > > > > > > old consumer's retirement. > > > > > > > > > > > > --Vahid > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Joel Koshy <jjkosh...@gmail.com> > > > > > > To: "dev@kafka.apache.org" <dev@kafka.apache.org> > > > > > > Date: 01/05/2017 06:55 PM > > > > > > Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation > > > > > > > > > > > > > > > > > > > > > > > > While I realize this only marks the old consumer as deprecated > and > > > not > > > > a > > > > > > complete removal, I agree that it is somewhat premature to do > this > > > > prior > > > > > > to > > > > > > having a migration process implemented. Onur has described this > in > > > > detail > > > > > > in the earlier thread: http://markmail.org/message/ > > ekv352zy7xttco5s > > > > and > > > > > > I'm > > > > > > surprised that more companies aren't affected by (or aware of?) > the > > > > > issue. > > > > > > > > > > > > On Thu, Jan 5, 2017 at 4:40 PM, radai < > radai.rosenbl...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > I cant speak for anyone else, but a rolling upgrade is > definitely > > > how > > > > > we > > > > > > > (LinkedIn) will do the migration. > > > > > > > > > > > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira < > g...@confluent.io> > > > > > wrote: > > > > > > > > > > > > > > > it sounds good to have > > > > > > > > it, but that's probably not how people will end up migrati > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
