Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-10 Thread Ewen Cheslack-Postava 
e there are a few consuming applications for which
> > the
> > > > current shut-down and restart approach to migration will suffice, I
> > doubt
> > > > we will be able to do this for majority of services that are outside
> > our
> > > > direct control. Given that a seamless migration is a pre-req for us
> to
> > > > switch to the new consumer widely (there are a few use-cases already
> on
> > > it)
> > > > it is something that we (LinkedIn) will likely implement although we
> > > > haven't done further brainstorming/improvements beyond what was
> > proposed
> > > in
> > > > the other deprecation thread.
> > > >
> > > >
> > > > > In the meantime, we have this suboptimal situation where the old
> > > > consumers
> > > > > are close to unmaintained even though we don't say it outright:
> they
> > > > don't
> > > >
> > > > get new features (basic things like security are missing) and bug
> fixes
> > > are
> > > > > rare. In practice, the old clients have been deprecated a while
> back,
> > > we
> > > > >
> > > >
> > > > Agreed that it is suboptimal, but the reality is that LI and I think
> a
> > > few
> > > > other companies are still to a large extent on the old consumer and
> > will
> > > be
> > > > for at least a good part of this year. This does mean that we have
> the
> > > > overhead of maintaining some internal workarounds for the old
> consumer.
> > > >
> > > >
> > > > > just haven't made it official. This proposal is about rectifying
> that
> > > so
> > > > > that we communicate our intentions to our users more clearly. As
> > Vahid
> > > > > said, this KIP is not about changing how we maintain the existing
> > code.
> > > > >
> > > > > The KIP that proposes the removal of all the old clients will be
> more
> > > > > interesting, but it doesn't exist yet. :)
> > > > >
> > > >
> > > > Deprecating *after* providing a sound migration path still seems to
> be
> > > the
> > > > right thing to do but if there isn't any demand for it then maybe
> > that's
> > > a
> > > > reasonable compromise. I'm still surprised that more users aren't as
> > > > impacted by this and as mentioned earlier, it could be an issue of
> > > > awareness but I'm not sure that deprecating before a migration path
> is
> > in
> > > > place would be considered best-practice in raising awareness.
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > > >
> > > >
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > > > vahidhashem...@us.ibm.com
> > > > > > wrote:
> > > > >
> > > > > > One thing that probably needs some clarification is what is
> implied
> > > by
> > > > > > "deprecated" in the Kafka project.
> > > > > > I googled it a bit and it doesn't seem that deprecation
> > > conventionally
> > > > > > implies termination of support (or anything that could negatively
> > > > impact
> > > > > > existing users). That's my interpretation too.
> > > > > > It would be good to know if Kafka follows a different
> > interpretation
> > > of
> > > > > > the term.
> > > > > >
> > > > > > If my understanding of the term is correct, since we are not yet
> > > > > targeting
> > > > > > a certain major release in which the old consumer will be
> removed,
> > I
> > > > > don't
> > > > > > see any harm in marking it as deprecated.
> > > > > > There will be enough time to plan and implement the migration, if
> > the
> > > > > > community decides that's the way to go, before phasing it out.
> > > > > >
> > > > > > At the minimum new Kafka users will pick the Java consumer
> without
> > > any
> > > > > > confusion. And existing users will know that Kafka is preparing
> for
> > > the
> > > > > > old consumer's retirement.
> > > > > >
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From:   Joel Koshy <jjkosh...@gmail.com>
> > > > > > To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> > > > > > Date:   01/05/2017 06:55 PM
> > > > > > Subject:Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > > > >
> > > > > >
> > > > > >
> > > > > > While I realize this only marks the old consumer as deprecated
> and
> > > not
> > > > a
> > > > > > complete removal, I agree that it is somewhat premature to do
> this
> > > > prior
> > > > > > to
> > > > > > having a migration process implemented. Onur has described this
> in
> > > > detail
> > > > > > in the earlier thread: http://markmail.org/message/
> > ekv352zy7xttco5s
> > > > and
> > > > > > I'm
> > > > > > surprised that more companies aren't affected by (or aware of?)
> the
> > > > > issue.
> > > > > >
> > > > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <
> radai.rosenbl...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > I cant speak for anyone else, but a rolling upgrade is
> definitely
> > > how
> > > > > we
> > > > > > > (LinkedIn) will do the migration.
> > > > > > >
> > > > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <
> g...@confluent.io>
> > > > > wrote:
> > > > > > >
> > > > > > > > it sounds good to have
> > > > > > > > it, but that's probably not how people will end up migrati
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-10 Thread Ismael Juma 
t;
> > >
> > > Deprecating *after* providing a sound migration path still seems to be
> > the
> > > right thing to do but if there isn't any demand for it then maybe
> that's
> > a
> > > reasonable compromise. I'm still surprised that more users aren't as
> > > impacted by this and as mentioned earlier, it could be an issue of
> > > awareness but I'm not sure that deprecating before a migration path is
> in
> > > place would be considered best-practice in raising awareness.
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > >
> > >
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > > vahidhashem...@us.ibm.com
> > > > > wrote:
> > > >
> > > > > One thing that probably needs some clarification is what is implied
> > by
> > > > > "deprecated" in the Kafka project.
> > > > > I googled it a bit and it doesn't seem that deprecation
> > conventionally
> > > > > implies termination of support (or anything that could negatively
> > > impact
> > > > > existing users). That's my interpretation too.
> > > > > It would be good to know if Kafka follows a different
> interpretation
> > of
> > > > > the term.
> > > > >
> > > > > If my understanding of the term is correct, since we are not yet
> > > > targeting
> > > > > a certain major release in which the old consumer will be removed,
> I
> > > > don't
> > > > > see any harm in marking it as deprecated.
> > > > > There will be enough time to plan and implement the migration, if
> the
> > > > > community decides that's the way to go, before phasing it out.
> > > > >
> > > > > At the minimum new Kafka users will pick the Java consumer without
> > any
> > > > > confusion. And existing users will know that Kafka is preparing for
> > the
> > > > > old consumer's retirement.
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From:   Joel Koshy <jjkosh...@gmail.com>
> > > > > To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> > > > > Date:   01/05/2017 06:55 PM
> > > > > Subject:Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > > >
> > > > >
> > > > >
> > > > > While I realize this only marks the old consumer as deprecated and
> > not
> > > a
> > > > > complete removal, I agree that it is somewhat premature to do this
> > > prior
> > > > > to
> > > > > having a migration process implemented. Onur has described this in
> > > detail
> > > > > in the earlier thread: http://markmail.org/message/
> ekv352zy7xttco5s
> > > and
> > > > > I'm
> > > > > surprised that more companies aren't affected by (or aware of?) the
> > > > issue.
> > > > >
> > > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <radai.rosenbl...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I cant speak for anyone else, but a rolling upgrade is definitely
> > how
> > > > we
> > > > > > (LinkedIn) will do the migration.
> > > > > >
> > > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <g...@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > > it sounds good to have
> > > > > > > it, but that's probably not how people will end up migrati
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-10 Thread Joel Koshy 
 > > > The KIP that proposes the removal of all the old clients will be more
> > > > interesting, but it doesn't exist yet. :)
> > > >
> > >
> > > Deprecating *after* providing a sound migration path still seems to be
> > the
> > > right thing to do but if there isn't any demand for it then maybe
> that's
> > a
> > > reasonable compromise. I'm still surprised that more users aren't as
> > > impacted by this and as mentioned earlier, it could be an issue of
> > > awareness but I'm not sure that deprecating before a migration path is
> in
> > > place would be considered best-practice in raising awareness.
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > >
> > >
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > > vahidhashem...@us.ibm.com
> > > > > wrote:
> > > >
> > > > > One thing that probably needs some clarification is what is implied
> > by
> > > > > "deprecated" in the Kafka project.
> > > > > I googled it a bit and it doesn't seem that deprecation
> > conventionally
> > > > > implies termination of support (or anything that could negatively
> > > impact
> > > > > existing users). That's my interpretation too.
> > > > > It would be good to know if Kafka follows a different
> interpretation
> > of
> > > > > the term.
> > > > >
> > > > > If my understanding of the term is correct, since we are not yet
> > > > targeting
> > > > > a certain major release in which the old consumer will be removed,
> I
> > > > don't
> > > > > see any harm in marking it as deprecated.
> > > > > There will be enough time to plan and implement the migration, if
> the
> > > > > community decides that's the way to go, before phasing it out.
> > > > >
> > > > > At the minimum new Kafka users will pick the Java consumer without
> > any
> > > > > confusion. And existing users will know that Kafka is preparing for
> > the
> > > > > old consumer's retirement.
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From:   Joel Koshy <jjkosh...@gmail.com>
> > > > > To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> > > > > Date:   01/05/2017 06:55 PM
> > > > > Subject:Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > > >
> > > > >
> > > > >
> > > > > While I realize this only marks the old consumer as deprecated and
> > not
> > > a
> > > > > complete removal, I agree that it is somewhat premature to do this
> > > prior
> > > > > to
> > > > > having a migration process implemented. Onur has described this in
> > > detail
> > > > > in the earlier thread: http://markmail.org/message/ek
> v352zy7xttco5s
> > > and
> > > > > I'm
> > > > > surprised that more companies aren't affected by (or aware of?) the
> > > > issue.
> > > > >
> > > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <radai.rosenbl...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I cant speak for anyone else, but a rolling upgrade is definitely
> > how
> > > > we
> > > > > > (LinkedIn) will do the migration.
> > > > > >
> > > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <g...@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > > it sounds good to have
> > > > > > > it, but that's probably not how people will end up migrati
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-10 Thread Renu Tewari 
ermination of support (or anything that could negatively
> > impact
> > > > existing users). That's my interpretation too.
> > > > It would be good to know if Kafka follows a different interpretation
> of
> > > > the term.
> > > >
> > > > If my understanding of the term is correct, since we are not yet
> > > targeting
> > > > a certain major release in which the old consumer will be removed, I
> > > don't
> > > > see any harm in marking it as deprecated.
> > > > There will be enough time to plan and implement the migration, if the
> > > > community decides that's the way to go, before phasing it out.
> > > >
> > > > At the minimum new Kafka users will pick the Java consumer without
> any
> > > > confusion. And existing users will know that Kafka is preparing for
> the
> > > > old consumer's retirement.
> > > >
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From:   Joel Koshy <jjkosh...@gmail.com>
> > > > To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> > > > Date:   01/05/2017 06:55 PM
> > > > Subject:Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > >
> > > >
> > > >
> > > > While I realize this only marks the old consumer as deprecated and
> not
> > a
> > > > complete removal, I agree that it is somewhat premature to do this
> > prior
> > > > to
> > > > having a migration process implemented. Onur has described this in
> > detail
> > > > in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s
> > and
> > > > I'm
> > > > surprised that more companies aren't affected by (or aware of?) the
> > > issue.
> > > >
> > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <radai.rosenbl...@gmail.com>
> > > wrote:
> > > >
> > > > > I cant speak for anyone else, but a rolling upgrade is definitely
> how
> > > we
> > > > > (LinkedIn) will do the migration.
> > > > >
> > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <g...@confluent.io>
> > > wrote:
> > > > >
> > > > > > it sounds good to have
> > > > > > it, but that's probably not how people will end up migrati
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-10 Thread Ismael Juma 
rt approach to migration will suffice, I
> doubt
> > > we will be able to do this for majority of services that are outside
> our
> > > direct control. Given that a seamless migration is a pre-req for us to
> > > switch to the new consumer widely (there are a few use-cases already on
> > it)
> > > it is something that we (LinkedIn) will likely implement although we
> > > haven't done further brainstorming/improvements beyond what was
> proposed
> > in
> > > the other deprecation thread.
> > >
> > >
> > > > In the meantime, we have this suboptimal situation where the old
> > > consumers
> > > > are close to unmaintained even though we don't say it outright: they
> > > don't
> > >
> > > get new features (basic things like security are missing) and bug fixes
> > are
> > > > rare. In practice, the old clients have been deprecated a while back,
> > we
> > > >
> > >
> > > Agreed that it is suboptimal, but the reality is that LI and I think a
> > few
> > > other companies are still to a large extent on the old consumer and
> will
> > be
> > > for at least a good part of this year. This does mean that we have the
> > > overhead of maintaining some internal workarounds for the old consumer.
> > >
> > >
> > > > just haven't made it official. This proposal is about rectifying that
> > so
> > > > that we communicate our intentions to our users more clearly. As
> Vahid
> > > > said, this KIP is not about changing how we maintain the existing
> code.
> > > >
> > > > The KIP that proposes the removal of all the old clients will be more
> > > > interesting, but it doesn't exist yet. :)
> > > >
> > >
> > > Deprecating *after* providing a sound migration path still seems to be
> > the
> > > right thing to do but if there isn't any demand for it then maybe
> that's
> > a
> > > reasonable compromise. I'm still surprised that more users aren't as
> > > impacted by this and as mentioned earlier, it could be an issue of
> > > awareness but I'm not sure that deprecating before a migration path is
> in
> > > place would be considered best-practice in raising awareness.
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > >
> > >
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > > vahidhashem...@us.ibm.com
> > > > > wrote:
> > > >
> > > > > One thing that probably needs some clarification is what is implied
> > by
> > > > > "deprecated" in the Kafka project.
> > > > > I googled it a bit and it doesn't seem that deprecation
> > conventionally
> > > > > implies termination of support (or anything that could negatively
> > > impact
> > > > > existing users). That's my interpretation too.
> > > > > It would be good to know if Kafka follows a different
> interpretation
> > of
> > > > > the term.
> > > > >
> > > > > If my understanding of the term is correct, since we are not yet
> > > > targeting
> > > > > a certain major release in which the old consumer will be removed,
> I
> > > > don't
> > > > > see any harm in marking it as deprecated.
> > > > > There will be enough time to plan and implement the migration, if
> the
> > > > > community decides that's the way to go, before phasing it out.
> > > > >
> > > > > At the minimum new Kafka users will pick the Java consumer without
> > any
> > > > > confusion. And existing users will know that Kafka is preparing for
> > the
> > > > > old consumer's retirement.
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From:   Joel Koshy <jjkosh...@gmail.com>
> > > > > To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> > > > > Date:   01/05/2017 06:55 PM
> > > > > Subject:Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > > >
> > > > >
> > > > >
> > > > > While I realize this only marks the old consumer as deprecated and
> > not
> > > a
> > > > > complete removal, I agree that it is somewhat premature to do this
> > > prior
> > > > > to
> > > > > having a migration process implemented. Onur has described this in
> > > detail
> > > > > in the earlier thread: http://markmail.org/message/ek
> v352zy7xttco5s
> > > and
> > > > > I'm
> > > > > surprised that more companies aren't affected by (or aware of?) the
> > > > issue.
> > > > >
> > > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <radai.rosenbl...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I cant speak for anyone else, but a rolling upgrade is definitely
> > how
> > > > we
> > > > > > (LinkedIn) will do the migration.
> > > > > >
> > > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <g...@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > > it sounds good to have
> > > > > > > it, but that's probably not how people will end up migrati
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-09 Thread Ewen Cheslack-Postava 
> said, this KIP is not about changing how we maintain the existing code.
> > >
> > > The KIP that proposes the removal of all the old clients will be more
> > > interesting, but it doesn't exist yet. :)
> > >
> >
> > Deprecating *after* providing a sound migration path still seems to be
> the
> > right thing to do but if there isn't any demand for it then maybe that's
> a
> > reasonable compromise. I'm still surprised that more users aren't as
> > impacted by this and as mentioned earlier, it could be an issue of
> > awareness but I'm not sure that deprecating before a migration path is in
> > place would be considered best-practice in raising awareness.
> >
> > Thanks,
> >
> > Joel
> >
> >
> >
> > >
> > > Ismael
> > >
> > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com
> > > > wrote:
> > >
> > > > One thing that probably needs some clarification is what is implied
> by
> > > > "deprecated" in the Kafka project.
> > > > I googled it a bit and it doesn't seem that deprecation
> conventionally
> > > > implies termination of support (or anything that could negatively
> > impact
> > > > existing users). That's my interpretation too.
> > > > It would be good to know if Kafka follows a different interpretation
> of
> > > > the term.
> > > >
> > > > If my understanding of the term is correct, since we are not yet
> > > targeting
> > > > a certain major release in which the old consumer will be removed, I
> > > don't
> > > > see any harm in marking it as deprecated.
> > > > There will be enough time to plan and implement the migration, if the
> > > > community decides that's the way to go, before phasing it out.
> > > >
> > > > At the minimum new Kafka users will pick the Java consumer without
> any
> > > > confusion. And existing users will know that Kafka is preparing for
> the
> > > > old consumer's retirement.
> > > >
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From:   Joel Koshy <jjkosh...@gmail.com>
> > > > To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> > > > Date:   01/05/2017 06:55 PM
> > > > Subject:Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > >
> > > >
> > > >
> > > > While I realize this only marks the old consumer as deprecated and
> not
> > a
> > > > complete removal, I agree that it is somewhat premature to do this
> > prior
> > > > to
> > > > having a migration process implemented. Onur has described this in
> > detail
> > > > in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s
> > and
> > > > I'm
> > > > surprised that more companies aren't affected by (or aware of?) the
> > > issue.
> > > >
> > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <radai.rosenbl...@gmail.com>
> > > wrote:
> > > >
> > > > > I cant speak for anyone else, but a rolling upgrade is definitely
> how
> > > we
> > > > > (LinkedIn) will do the migration.
> > > > >
> > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <g...@confluent.io>
> > > wrote:
> > > > >
> > > > > > it sounds good to have
> > > > > > it, but that's probably not how people will end up migrati
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-09 Thread Ismael Juma 
Hi Joel,

Great to hear that LinkedIn is likely to implement KAFKA-4513. :)

Yes, the KIP as it stands is a compromise given the situation. We could
push the deprecation to the subsequent release: likely to be 0.11.0.0 since
there are a number of KIPs that require message format changes. This would
mean that the old consumers would not be removed before 0.12.0.0 (the major
release after 0.11.0.0). Would that work better for you all?

Ismael

On Tue, Jan 10, 2017 at 12:52 AM, Joel Koshy <jjkosh...@gmail.com> wrote:

> >
> >
> > The ideal scenario would be for us to provide a tool for no downtime
> > migration as discussed in the original thread (I filed
> > https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
> > discussion). There are a few issues, however:
> >
> >- There doesn't seem to be much demand for it (outside of LinkedIn, at
> >least)
> >- No-one is working on it or has indicated that they are planning to
> >work on it
> >- It's a non-trivial change and it requires a good amount of testing
> to
> >ensure it works as expected
> >
>
> For LinkedIn: while there are a few consuming applications for which the
> current shut-down and restart approach to migration will suffice, I doubt
> we will be able to do this for majority of services that are outside our
> direct control. Given that a seamless migration is a pre-req for us to
> switch to the new consumer widely (there are a few use-cases already on it)
> it is something that we (LinkedIn) will likely implement although we
> haven't done further brainstorming/improvements beyond what was proposed in
> the other deprecation thread.
>
>
> > In the meantime, we have this suboptimal situation where the old
> consumers
> > are close to unmaintained even though we don't say it outright: they
> don't
>
> get new features (basic things like security are missing) and bug fixes are
> > rare. In practice, the old clients have been deprecated a while back, we
> >
>
> Agreed that it is suboptimal, but the reality is that LI and I think a few
> other companies are still to a large extent on the old consumer and will be
> for at least a good part of this year. This does mean that we have the
> overhead of maintaining some internal workarounds for the old consumer.
>
>
> > just haven't made it official. This proposal is about rectifying that so
> > that we communicate our intentions to our users more clearly. As Vahid
> > said, this KIP is not about changing how we maintain the existing code.
> >
> > The KIP that proposes the removal of all the old clients will be more
> > interesting, but it doesn't exist yet. :)
> >
>
> Deprecating *after* providing a sound migration path still seems to be the
> right thing to do but if there isn't any demand for it then maybe that's a
> reasonable compromise. I'm still surprised that more users aren't as
> impacted by this and as mentioned earlier, it could be an issue of
> awareness but I'm not sure that deprecating before a migration path is in
> place would be considered best-practice in raising awareness.
>
> Thanks,
>
> Joel
>
>
>
> >
> > Ismael
> >
> > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com
> > > wrote:
> >
> > > One thing that probably needs some clarification is what is implied by
> > > "deprecated" in the Kafka project.
> > > I googled it a bit and it doesn't seem that deprecation conventionally
> > > implies termination of support (or anything that could negatively
> impact
> > > existing users). That's my interpretation too.
> > > It would be good to know if Kafka follows a different interpretation of
> > > the term.
> > >
> > > If my understanding of the term is correct, since we are not yet
> > targeting
> > > a certain major release in which the old consumer will be removed, I
> > don't
> > > see any harm in marking it as deprecated.
> > > There will be enough time to plan and implement the migration, if the
> > > community decides that's the way to go, before phasing it out.
> > >
> > > At the minimum new Kafka users will pick the Java consumer without any
> > > confusion. And existing users will know that Kafka is preparing for the
> > > old consumer's retirement.
> > >
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From:   Joel Koshy <jjkosh...@gmail.com>
> > > To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> > > Date:   01/05/2017 06:55

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-09 Thread Joel Koshy 
>
>
> The ideal scenario would be for us to provide a tool for no downtime
> migration as discussed in the original thread (I filed
> https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
> discussion). There are a few issues, however:
>
>- There doesn't seem to be much demand for it (outside of LinkedIn, at
>least)
>- No-one is working on it or has indicated that they are planning to
>work on it
>- It's a non-trivial change and it requires a good amount of testing to
>ensure it works as expected
>

For LinkedIn: while there are a few consuming applications for which the
current shut-down and restart approach to migration will suffice, I doubt
we will be able to do this for majority of services that are outside our
direct control. Given that a seamless migration is a pre-req for us to
switch to the new consumer widely (there are a few use-cases already on it)
it is something that we (LinkedIn) will likely implement although we
haven't done further brainstorming/improvements beyond what was proposed in
the other deprecation thread.


> In the meantime, we have this suboptimal situation where the old consumers
> are close to unmaintained even though we don't say it outright: they don't

get new features (basic things like security are missing) and bug fixes are
> rare. In practice, the old clients have been deprecated a while back, we
>

Agreed that it is suboptimal, but the reality is that LI and I think a few
other companies are still to a large extent on the old consumer and will be
for at least a good part of this year. This does mean that we have the
overhead of maintaining some internal workarounds for the old consumer.


> just haven't made it official. This proposal is about rectifying that so
> that we communicate our intentions to our users more clearly. As Vahid
> said, this KIP is not about changing how we maintain the existing code.
>
> The KIP that proposes the removal of all the old clients will be more
> interesting, but it doesn't exist yet. :)
>

Deprecating *after* providing a sound migration path still seems to be the
right thing to do but if there isn't any demand for it then maybe that's a
reasonable compromise. I'm still surprised that more users aren't as
impacted by this and as mentioned earlier, it could be an issue of
awareness but I'm not sure that deprecating before a migration path is in
place would be considered best-practice in raising awareness.

Thanks,

Joel



>
> Ismael
>
> On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com
> > wrote:
>
> > One thing that probably needs some clarification is what is implied by
> > "deprecated" in the Kafka project.
> > I googled it a bit and it doesn't seem that deprecation conventionally
> > implies termination of support (or anything that could negatively impact
> > existing users). That's my interpretation too.
> > It would be good to know if Kafka follows a different interpretation of
> > the term.
> >
> > If my understanding of the term is correct, since we are not yet
> targeting
> > a certain major release in which the old consumer will be removed, I
> don't
> > see any harm in marking it as deprecated.
> > There will be enough time to plan and implement the migration, if the
> > community decides that's the way to go, before phasing it out.
> >
> > At the minimum new Kafka users will pick the Java consumer without any
> > confusion. And existing users will know that Kafka is preparing for the
> > old consumer's retirement.
> >
> > --Vahid
> >
> >
> >
> >
> > From:   Joel Koshy <jjkosh...@gmail.com>
> > To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> > Date:   01/05/2017 06:55 PM
> > Subject:Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> >
> >
> >
> > While I realize this only marks the old consumer as deprecated and not a
> > complete removal, I agree that it is somewhat premature to do this prior
> > to
> > having a migration process implemented. Onur has described this in detail
> > in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s and
> > I'm
> > surprised that more companies aren't affected by (or aware of?) the
> issue.
> >
> > On Thu, Jan 5, 2017 at 4:40 PM, radai <radai.rosenbl...@gmail.com>
> wrote:
> >
> > > I cant speak for anyone else, but a rolling upgrade is definitely how
> we
> > > (LinkedIn) will do the migration.
> > >
> > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <g...@confluent.io>
> wrote:
> > >
> > > > it sounds good to have
> > > > it, but that's probably not how people will end up migrati
> > > >
> > >
> >
> >
> >
> >
> >
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-06 Thread Ismael Juma 
Hi all,

Thanks for the feedback. A few thoughts follow.

The ideal scenario would be for us to provide a tool for no downtime
migration as discussed in the original thread (I filed
https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
discussion). There are a few issues, however:

   - There doesn't seem to be much demand for it (outside of LinkedIn, at
   least)
   - No-one is working on it or has indicated that they are planning to
   work on it
   - It's a non-trivial change and it requires a good amount of testing to
   ensure it works as expected

I suggested the KIP to raise awareness. Maybe there's more demand than we
think and/or someone is planning to work on it. The latter, in particular,
would be great news.

In the meantime, we have this suboptimal situation where the old consumers
are close to unmaintained even though we don't say it outright: they don't
get new features (basic things like security are missing) and bug fixes are
rare. In practice, the old clients have been deprecated a while back, we
just haven't made it official. This proposal is about rectifying that so
that we communicate our intentions to our users more clearly. As Vahid
said, this KIP is not about changing how we maintain the existing code.

The KIP that proposes the removal of all the old clients will be more
interesting, but it doesn't exist yet. :)

Ismael

On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <vahidhashem...@us.ibm.com
> wrote:

> One thing that probably needs some clarification is what is implied by
> "deprecated" in the Kafka project.
> I googled it a bit and it doesn't seem that deprecation conventionally
> implies termination of support (or anything that could negatively impact
> existing users). That's my interpretation too.
> It would be good to know if Kafka follows a different interpretation of
> the term.
>
> If my understanding of the term is correct, since we are not yet targeting
> a certain major release in which the old consumer will be removed, I don't
> see any harm in marking it as deprecated.
> There will be enough time to plan and implement the migration, if the
> community decides that's the way to go, before phasing it out.
>
> At the minimum new Kafka users will pick the Java consumer without any
> confusion. And existing users will know that Kafka is preparing for the
> old consumer's retirement.
>
> --Vahid
>
>
>
>
> From:   Joel Koshy <jjkosh...@gmail.com>
> To:     "dev@kafka.apache.org" <dev@kafka.apache.org>
> Date:   01/05/2017 06:55 PM
> Subject:Re: [DISCUSS] KIP-109: Old Consumer Deprecation
>
>
>
> While I realize this only marks the old consumer as deprecated and not a
> complete removal, I agree that it is somewhat premature to do this prior
> to
> having a migration process implemented. Onur has described this in detail
> in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s and
> I'm
> surprised that more companies aren't affected by (or aware of?) the issue.
>
> On Thu, Jan 5, 2017 at 4:40 PM, radai <radai.rosenbl...@gmail.com> wrote:
>
> > I cant speak for anyone else, but a rolling upgrade is definitely how we
> > (LinkedIn) will do the migration.
> >
> > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <g...@confluent.io> wrote:
> >
> > > it sounds good to have
> > > it, but that's probably not how people will end up migrati
> > >
> >
>
>
>
>
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-05 Thread Vahid S Hashemian 
One thing that probably needs some clarification is what is implied by 
"deprecated" in the Kafka project.
I googled it a bit and it doesn't seem that deprecation conventionally 
implies termination of support (or anything that could negatively impact 
existing users). That's my interpretation too.
It would be good to know if Kafka follows a different interpretation of 
the term.

If my understanding of the term is correct, since we are not yet targeting 
a certain major release in which the old consumer will be removed, I don't 
see any harm in marking it as deprecated.
There will be enough time to plan and implement the migration, if the 
community decides that's the way to go, before phasing it out.

At the minimum new Kafka users will pick the Java consumer without any 
confusion. And existing users will know that Kafka is preparing for the 
old consumer's retirement.

--Vahid




From:   Joel Koshy <jjkosh...@gmail.com>
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Date:   01/05/2017 06:55 PM
Subject:    Re: [DISCUSS] KIP-109: Old Consumer Deprecation



While I realize this only marks the old consumer as deprecated and not a
complete removal, I agree that it is somewhat premature to do this prior 
to
having a migration process implemented. Onur has described this in detail
in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s and 
I'm
surprised that more companies aren't affected by (or aware of?) the issue.

On Thu, Jan 5, 2017 at 4:40 PM, radai <radai.rosenbl...@gmail.com> wrote:

> I cant speak for anyone else, but a rolling upgrade is definitely how we
> (LinkedIn) will do the migration.
>
> On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <g...@confluent.io> wrote:
>
> > it sounds good to have
> > it, but that's probably not how people will end up migrati
> >
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-05 Thread Joel Koshy 
While I realize this only marks the old consumer as deprecated and not a
complete removal, I agree that it is somewhat premature to do this prior to
having a migration process implemented. Onur has described this in detail
in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s and I'm
surprised that more companies aren't affected by (or aware of?) the issue.

On Thu, Jan 5, 2017 at 4:40 PM, radai  wrote:

> I cant speak for anyone else, but a rolling upgrade is definitely how we
> (LinkedIn) will do the migration.
>
> On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira  wrote:
>
> > it sounds good to have
> > it, but that's probably not how people will end up migrati
> >
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-05 Thread radai 
I cant speak for anyone else, but a rolling upgrade is definitely how we
(LinkedIn) will do the migration.

On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira  wrote:

> it sounds good to have
> it, but that's probably not how people will end up migrati
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-05 Thread Gwen Shapira 
Since the APIs are super different, I expect migrating from the old to
the new consumer will involve some re-write of the app that does the
consuming. In most such cases, the upgrade path involves running both
versions side-by-side for a while, validating results and then
retiring the old version. Sometimes migration of offsets is needed and
Grant published a tool for that a while back.

Having a rolling upgrade plan between both APIs is pretty involved and
I'm not sure there's a real demand for it (i.e. it sounds good to have
it, but that's probably not how people will end up migrating).

Gwen

On Thu, Jan 5, 2017 at 3:42 PM, radai  wrote:
> im all for (working towards) getting rid of old code, but there's still no
> solid migration path - you'll be "stranding" users on deprecated, no longer
> maintained code with no "safe" way out that does not involve downtime
> (specifically old and new consumers cannot correctly divide up partitions
> between themselves if both operate within the same group on the same topic).
>
> On Thu, Jan 5, 2017 at 3:10 PM, Gwen Shapira  wrote:
>
>> Very strong support from me too :)
>>
>> On Thu, Jan 5, 2017 at 12:09 PM, Vahid S Hashemian
>>  wrote:
>> > Hi all,
>> >
>> > There was some discussion recently on deprecating the old consumer (
>> > https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
>> > Ismael suggested to cover the discussion and voting of major deprecations
>> > like this under a KIP.
>> >
>> > So I started KIP-109 (
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 109%3A+Old+Consumer+Deprecation
>> > ) and look forward to your feedback and comments.
>> >
>> > We'd like to implement this deprecation in the upcoming 0.10.2.0 release.
>> >
>> > Thanks.
>> > --Vahid
>> >
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-05 Thread radai 
im all for (working towards) getting rid of old code, but there's still no
solid migration path - you'll be "stranding" users on deprecated, no longer
maintained code with no "safe" way out that does not involve downtime
(specifically old and new consumers cannot correctly divide up partitions
between themselves if both operate within the same group on the same topic).

On Thu, Jan 5, 2017 at 3:10 PM, Gwen Shapira  wrote:

> Very strong support from me too :)
>
> On Thu, Jan 5, 2017 at 12:09 PM, Vahid S Hashemian
>  wrote:
> > Hi all,
> >
> > There was some discussion recently on deprecating the old consumer (
> > https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
> > Ismael suggested to cover the discussion and voting of major deprecations
> > like this under a KIP.
> >
> > So I started KIP-109 (
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 109%3A+Old+Consumer+Deprecation
> > ) and look forward to your feedback and comments.
> >
> > We'd like to implement this deprecation in the upcoming 0.10.2.0 release.
> >
> > Thanks.
> > --Vahid
> >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-05 Thread Gwen Shapira 
Very strong support from me too :)

On Thu, Jan 5, 2017 at 12:09 PM, Vahid S Hashemian
 wrote:
> Hi all,
>
> There was some discussion recently on deprecating the old consumer (
> https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
> Ismael suggested to cover the discussion and voting of major deprecations
> like this under a KIP.
>
> So I started KIP-109 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-109%3A+Old+Consumer+Deprecation
> ) and look forward to your feedback and comments.
>
> We'd like to implement this deprecation in the upcoming 0.10.2.0 release.
>
> Thanks.
> --Vahid
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-05 Thread Tom Crayford 
It's not time for voting, but a huge +1 from me.

On Thu, Jan 5, 2017 at 8:25 PM, Vahid S Hashemian <vahidhashem...@us.ibm.com
> wrote:

> Thanks Ismael. I added that to the KIP.
>
>
>
>
> From:   Ismael Juma <ism...@juma.me.uk>
> To: dev@kafka.apache.org
> Date:   01/05/2017 12:16 PM
> Subject:    Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> Sent by:isma...@gmail.com
>
>
>
> Thanks Vahid, +1 (predictably). Worth mentioning in the KIP that
> compatibility with older brokers (0.10.0 and later) is another feature
> that
> will only be supported by the Java consumer.
>
> Ismael
>
> On 5 Jan 2017 8:10 pm, "Vahid S Hashemian" <vahidhashem...@us.ibm.com>
> wrote:
>
> > Hi all,
> >
> > There was some discussion recently on deprecating the old consumer (
> > https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
> > Ismael suggested to cover the discussion and voting of major
> deprecations
> > like this under a KIP.
> >
> > So I started KIP-109 (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-109%3A+Old+Consumer+
> > Deprecation
> > ) and look forward to your feedback and comments.
> >
> > We'd like to implement this deprecation in the upcoming 0.10.2.0
> release.
> >
> > Thanks.
> > --Vahid
> >
> >
>
>
>
>
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-05 Thread Vahid S Hashemian 
Thanks Ismael. I added that to the KIP.




From:   Ismael Juma <ism...@juma.me.uk>
To: dev@kafka.apache.org
Date:   01/05/2017 12:16 PM
Subject:        Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Sent by:isma...@gmail.com



Thanks Vahid, +1 (predictably). Worth mentioning in the KIP that
compatibility with older brokers (0.10.0 and later) is another feature 
that
will only be supported by the Java consumer.

Ismael

On 5 Jan 2017 8:10 pm, "Vahid S Hashemian" <vahidhashem...@us.ibm.com>
wrote:

> Hi all,
>
> There was some discussion recently on deprecating the old consumer (
> https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
> Ismael suggested to cover the discussion and voting of major 
deprecations
> like this under a KIP.
>
> So I started KIP-109 (
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-109%3A+Old+Consumer+
> Deprecation
> ) and look forward to your feedback and comments.
>
> We'd like to implement this deprecation in the upcoming 0.10.2.0 
release.
>
> Thanks.
> --Vahid
>
>

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-05 Thread Ismael Juma 
Thanks Vahid, +1 (predictably). Worth mentioning in the KIP that
compatibility with older brokers (0.10.0 and later) is another feature that
will only be supported by the Java consumer.

Ismael

On 5 Jan 2017 8:10 pm, "Vahid S Hashemian" 
wrote:

> Hi all,
>
> There was some discussion recently on deprecating the old consumer (
> https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
> Ismael suggested to cover the discussion and voting of major deprecations
> like this under a KIP.
>
> So I started KIP-109 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-109%3A+Old+Consumer+
> Deprecation
> ) and look forward to your feedback and comments.
>
> We'd like to implement this deprecation in the upcoming 0.10.2.0 release.
>
> Thanks.
> --Vahid
>
>