Hi! I agree that it’s very confusing if you explicitly specify the topics that are to be confusing and what happens is different.
I would almost consider this to be a bug , can’t see any reasonable use case just hard to debug problems . Having an option would be a good start but I would rather treat this as a bug. Gyula On Wed, 13 Feb 2019 at 18:27, Feng LI <nemoking...@gmail.com> wrote: > Hello there, > > I’m just wondering if there are real world use cases for maintaining this > default behavior. It’s a bit counter intuitive and sometimes results in > serious production issues. ( We had a similar issue when changing the topic > name, and resulting reading every message twice - both from the old one and > from the new). > > Cheers, > Feng > Le mer. 13 févr. 2019 à 17:56, Tzu-Li (Gordon) Tai <tzuli...@apache.org> a > écrit : > > > Hi, > > > > Partition offsets stored in state will always be respected when the > > consumer is restored from checkpoints / savepoints. > > AFAIK, this seems to have been the behaviour for quite some time now > (since > > FlinkKafkaConsumer08). > > > > I think in the past there were some discussion to at least allow some way > > to ignore restored partition offsets. > > One way to enable this is to filter the restored partition offsets based > on > > the configured list of specified topics / topic regex pattern in the > > current execution. This should work, since this can only be modified when > > restoring from savepoints (i.e. manual restores). > > To avoid breaking the current behaviour, we can maybe add a > > `filterRestoredPartitionOffsetState()` configuration on the consumer, > which > > by default is disabled to match the current behaviour. > > > > What do you think? > > > > Cheers, > > Gordon > > > > On Wed, Feb 13, 2019 at 11:59 PM Gyula Fóra <gyula.f...@gmail.com> > wrote: > > > > > Hi! > > > > > > I have run into a weird issue which I could have sworn that it wouldnt > > > happen :D > > > I feel there was a discussion about this in the past but maybe im > wrong, > > > but I hope someone can point me to a ticket. > > > > > > Lets say you create a kafka consumer that consumes (t1,t2,t3), you > take a > > > savepoint and deploy a new version that only consumes (t1). > > > > > > The restore logic now still starts to consume (t1,t2,t3) which feels > very > > > unintuitive as those were explicitly removed from the list. It is also > > hard > > > to debug as the topics causing the problem are not defined anywhere in > > your > > > job, configs etc. > > > > > > Has anyone run into this issue? Should we change this default behaviour > > or > > > at least have an option to not do this? > > > > > > Cheers, > > > Gyula > > > > > >
