Thanks James. I had read your post and was planning to find it in order to share it here so you saved me some work. :)
Ismael On Fri, Nov 18, 2016 at 3:21 AM, James Cheng <wushuja...@gmail.com> wrote: > Sorry to self-plug, but I wrote a blog post that talks about this, with > respect to mirrormaker. I came to the same 3 solutions that Onur described. > > https://logallthethings.com/2016/10/07/mirrormaker- > gotchas-when-moving-from-the-old-consumer-to-the-new-consumer/ < > https://logallthethings.com/2016/10/07/mirrormaker- > gotchas-when-moving-from-the-old-consumer-to-the-new-consumer/> > > -James > > > On Nov 17, 2016, at 7:37 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > > Hi Onur, > > > > It is a good point that there is currently no out of the box solution for > > migrating from the old consumer to the new consumer where neither > downtime > > or duplicate consumption are acceptable. As I understand, this is > important > > for some of the usages at LinkedIn. Do you have any plans to tackle this > > issue? > > > > Jason, any thoughts on this? > > > > Ismael > > > > On Mon, Oct 31, 2016 at 11:03 PM, Onur Karaman < > > okara...@linkedin.com.invalid> wrote: > > > >> Does this make sense given that we still don't have a graceful migration > >> plan from the old to new consumer? > >> > >> Different suboptimal migration plans that I can think of are: > >> 1. shutdown all the old consumers of a group first and start them back > up > >> with the new consumer, causing downtime. > >> 2. have a mix of old and new consumers in the same group, causing > duplicate > >> partition ownership and consumption as each rebalance protocol ignores > the > >> other. > >> 3. form a brand new group for the new consumers doing the same work as > the > >> old consumer group, still causing duplicate partition ownership and > >> consumption across the two groups. > >> > >> On Mon, Oct 31, 2016 at 3:42 PM, Jun Rao <j...@confluent.io> wrote: > >> > >>> Starting to deprecate the old consumer in the next release seems like a > >>> good idea. > >>> > >>> Thanks, > >>> > >>> Jun > >>> > >>> On Tue, Oct 25, 2016 at 2:45 AM, Ismael Juma <ism...@juma.me.uk> > wrote: > >>> > >>>> Hi all, > >>>> > >>>> In 0.10.1.0, we removed the beta label from the new Java consumer > >>>> documentation and updated the various tools so that they can use the > >> new > >>>> consumer without having to pass the `--new-consumer` flag (more > >>>> specifically the new consumer is used if `bootstrap-server` is set). > >> More > >>>> details of the reasoning can be found in the original discuss thread: > >>>> http://search-hadoop.com/m/Kafka/uyzND1e4bUP1Rjq721 > >>>> > >>>> The old consumers don't have security or `offsetsForTimestamp` > (KIP-79) > >>>> support and the plan is to only add features to the new Java consumer. > >>> Even > >>>> so, the old consumers are a significant maintenance burden as they > >>>> duplicate protocol request/response classes (the SimpleConsumer > exposes > >>>> them in the public API sadly). I experienced this first hand most > >>> recently > >>>> while working on KIP-74. > >>>> > >>>> Given the above, I propose we deprecate the old consumers in trunk to > >>> nudge > >>>> users in the right direction. Users will have the 0.10.1.0 cycle to > >> start > >>>> migrating to the new Java consumer with no build warnings. Once they > >>>> upgrade to the next release (i.e. 0.10.2.0), users who are still using > >>> the > >>>> old consumers will get warnings at build time encouraging them to move > >> to > >>>> the new consumer, but everything will still work fine. > >>>> > >>>> In a future major release, the old consumers (along with the old > >>> producers) > >>>> will be removed. We will have a separate discuss/vote thread for that > >> to > >>>> make sure the time is right. > >>>> > >>>> Thoughts? > >>>> > >>>> Ismael > >>>> > >>> > >> > >
