Hi Onur, It is a good point that there is currently no out of the box solution for migrating from the old consumer to the new consumer where neither downtime or duplicate consumption are acceptable. As I understand, this is important for some of the usages at LinkedIn. Do you have any plans to tackle this issue?
Jason, any thoughts on this? Ismael On Mon, Oct 31, 2016 at 11:03 PM, Onur Karaman < okara...@linkedin.com.invalid> wrote: > Does this make sense given that we still don't have a graceful migration > plan from the old to new consumer? > > Different suboptimal migration plans that I can think of are: > 1. shutdown all the old consumers of a group first and start them back up > with the new consumer, causing downtime. > 2. have a mix of old and new consumers in the same group, causing duplicate > partition ownership and consumption as each rebalance protocol ignores the > other. > 3. form a brand new group for the new consumers doing the same work as the > old consumer group, still causing duplicate partition ownership and > consumption across the two groups. > > On Mon, Oct 31, 2016 at 3:42 PM, Jun Rao <j...@confluent.io> wrote: > > > Starting to deprecate the old consumer in the next release seems like a > > good idea. > > > > Thanks, > > > > Jun > > > > On Tue, Oct 25, 2016 at 2:45 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > > > Hi all, > > > > > > In 0.10.1.0, we removed the beta label from the new Java consumer > > > documentation and updated the various tools so that they can use the > new > > > consumer without having to pass the `--new-consumer` flag (more > > > specifically the new consumer is used if `bootstrap-server` is set). > More > > > details of the reasoning can be found in the original discuss thread: > > > http://search-hadoop.com/m/Kafka/uyzND1e4bUP1Rjq721 > > > > > > The old consumers don't have security or `offsetsForTimestamp` (KIP-79) > > > support and the plan is to only add features to the new Java consumer. > > Even > > > so, the old consumers are a significant maintenance burden as they > > > duplicate protocol request/response classes (the SimpleConsumer exposes > > > them in the public API sadly). I experienced this first hand most > > recently > > > while working on KIP-74. > > > > > > Given the above, I propose we deprecate the old consumers in trunk to > > nudge > > > users in the right direction. Users will have the 0.10.1.0 cycle to > start > > > migrating to the new Java consumer with no build warnings. Once they > > > upgrade to the next release (i.e. 0.10.2.0), users who are still using > > the > > > old consumers will get warnings at build time encouraging them to move > to > > > the new consumer, but everything will still work fine. > > > > > > In a future major release, the old consumers (along with the old > > producers) > > > will be removed. We will have a separate discuss/vote thread for that > to > > > make sure the time is right. > > > > > > Thoughts? > > > > > > Ismael > > > > > >
