Not sure it is worth doing, but a simple migration approach that avoids
*service* downtime could be as follows:
- Add a “migration mode” to the old consumer that disables its fetchers
and disables offset commits. i.e., the consumers rebalance and own
partitions but do basically nothing.
- So assuming the old consumer is already committing offsets to Kafka,
the process would be:
- Bounce the consumer group (still on the old consumer) with:
- Migration mode on
- consumer.timeout.ms -1
- Bounce the consumer group to switch to the new consumer
- i.e., effectively pause and resume the entire group without real
downtime of the services.
On Thu, Nov 17, 2016 at 7:30 PM, Ismael Juma <ism...@juma.me.uk> wrote:
> Thanks James. I had read your post and was planning to find it in order to
> share it here so you saved me some work. :)
>
> Ismael
>
> On Fri, Nov 18, 2016 at 3:21 AM, James Cheng <wushuja...@gmail.com> wrote:
>
> > Sorry to self-plug, but I wrote a blog post that talks about this, with
> > respect to mirrormaker. I came to the same 3 solutions that Onur
> described.
> >
> > https://logallthethings.com/2016/10/07/mirrormaker-
> > gotchas-when-moving-from-the-old-consumer-to-the-new-consumer/ <
> > https://logallthethings.com/2016/10/07/mirrormaker-
> > gotchas-when-moving-from-the-old-consumer-to-the-new-consumer/>
> >
> > -James
> >
> > > On Nov 17, 2016, at 7:37 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> > >
> > > Hi Onur,
> > >
> > > It is a good point that there is currently no out of the box solution
> for
> > > migrating from the old consumer to the new consumer where neither
> > downtime
> > > or duplicate consumption are acceptable. As I understand, this is
> > important
> > > for some of the usages at LinkedIn. Do you have any plans to tackle
> this
> > > issue?
> > >
> > > Jason, any thoughts on this?
> > >
> > > Ismael
> > >
> > > On Mon, Oct 31, 2016 at 11:03 PM, Onur Karaman <
> > > okara...@linkedin.com.invalid> wrote:
> > >
> > >> Does this make sense given that we still don't have a graceful
> migration
> > >> plan from the old to new consumer?
> > >>
> > >> Different suboptimal migration plans that I can think of are:
> > >> 1. shutdown all the old consumers of a group first and start them back
> > up
> > >> with the new consumer, causing downtime.
> > >> 2. have a mix of old and new consumers in the same group, causing
> > duplicate
> > >> partition ownership and consumption as each rebalance protocol ignores
> > the
> > >> other.
> > >> 3. form a brand new group for the new consumers doing the same work as
> > the
> > >> old consumer group, still causing duplicate partition ownership and
> > >> consumption across the two groups.
> > >>
> > >> On Mon, Oct 31, 2016 at 3:42 PM, Jun Rao <j...@confluent.io> wrote:
> > >>
> > >>> Starting to deprecate the old consumer in the next release seems
> like a
> > >>> good idea.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jun
> > >>>
> > >>> On Tue, Oct 25, 2016 at 2:45 AM, Ismael Juma <ism...@juma.me.uk>
> > wrote:
> > >>>
> > >>>> Hi all,
> > >>>>
> > >>>> In 0.10.1.0, we removed the beta label from the new Java consumer
> > >>>> documentation and updated the various tools so that they can use the
> > >> new
> > >>>> consumer without having to pass the `--new-consumer` flag (more
> > >>>> specifically the new consumer is used if `bootstrap-server` is set).
> > >> More
> > >>>> details of the reasoning can be found in the original discuss
> thread:
> > >>>> http://search-hadoop.com/m/Kafka/uyzND1e4bUP1Rjq721
> > >>>>
> > >>>> The old consumers don't have security or `offsetsForTimestamp`
> > (KIP-79)
> > >>>> support and the plan is to only add features to the new Java
> consumer.
> > >>> Even
> > >>>> so, the old consumers are a significant maintenance burden as they
> > >>>> duplicate protocol request/response classes (the SimpleConsumer
> > exposes
> > >>>> them in the public API sadly). I experienced this first hand most
> > >>> recently
> > >>>> while working on KIP-74.
> > >>>>
> > >>>> Given the above, I propose we deprecate the old consumers in trunk
> to
> > >>> nudge
> > >>>> users in the right direction. Users will have the 0.10.1.0 cycle to
> > >> start
> > >>>> migrating to the new Java consumer with no build warnings. Once they
> > >>>> upgrade to the next release (i.e. 0.10.2.0), users who are still
> using
> > >>> the
> > >>>> old consumers will get warnings at build time encouraging them to
> move
> > >> to
> > >>>> the new consumer, but everything will still work fine.
> > >>>>
> > >>>> In a future major release, the old consumers (along with the old
> > >>> producers)
> > >>>> will be removed. We will have a separate discuss/vote thread for
> that
> > >> to
> > >>>> make sure the time is right.
> > >>>>
> > >>>> Thoughts?
> > >>>>
> > >>>> Ismael
> > >>>>
> > >>>
> > >>
> >
> >
>
