Opened a Jira ticket: https://issues.apache.org/jira/browse/KAFKA-9824
On Mon, Apr 6, 2020 at 10:43 AM Nitay Kufert <nita...@ironsrc.com> wrote: > 2.3.1 - Both broker & clients upgrade > > On Sun, Apr 5, 2020 at 8:52 PM Ismael Juma <ism...@juma.me.uk> wrote: > >> Hi Nitay, >> >> What version were you running before the upgrade? And was this a broker >> upgrade? I would suggest filing a JIRA ticket, including as much >> information as possible and ideally consumer and broker logs from the >> period where the issue happened. >> >> Ismael >> >> On Sun, Apr 5, 2020, 10:13 AM Nitay Kufert <nita...@ironsrc.com> wrote: >> >> > Hey, >> > >> > We are using kafka streams across our tech stack for the last 2-3 years. >> > >> > Lately, we moved to kafka 2.4.1 and we started noticing a troubling >> > behaviour that we didn't see before: >> > >> > Without apparent reason, a specific partition on a specific consumer >> loses >> > its offset, with the following messages appearing on the consumer: >> > >> >> >> >> [Consumer clientId=consumer-XConsumerGroup4-2, groupId=XConsumerGroup4] >> >> Fetch offset 1222791071 is out of range for partition SomeTopic-48, >> >> resetting offset: >> >> >> > [Consumer clientId=consumer-XConsumerGroup4-2, groupId=XConsumerGroup4] >> >> Resetting offset for partition SomeTopic-48 to offset 1216430527.: >> > >> > >> > from the broker, the only thing i could find is: >> > >> >> BrokerId 1033 is no longer a coordinator for the group >> XConsumerGroup4. Proceeding cleanup for other alive groups >> >> >> >> >> > Another way to see the same thing, from our monitoring (DD): >> > [image: Screen Shot 2020-04-05 at 16.32.19.png] >> > >> > The recovery you are seeing is after I run partition offset reset >> manually >> > (using kafka-consumer-groups.sh --bootstrap-server localhost:9092 >> --topic >> > SomeTopic:57 --group XConsumerGroup4 --reset-offsets --to-datetime >> > '2020-04-02T07:00:00.000') >> > >> > Any idea what can be causing this? we have it happen to us at least 5 >> > times since the upgrade, and before that i don't remember it even >> happening >> > to us. >> > >> > Let me know if you need more data from me >> > Thanks >> > -- >> > >> > Nitay Kufert >> > Backend Team Leader >> > [image: ironSource] <http://www.ironsrc.com> >> > >> > email nita...@ironsrc.com >> > mobile +972-54-5480021 >> > fax +972-77-5448273 >> > skype nitay.kufert.ssa >> > 121 Menachem Begin St., Tel Aviv, Israel >> > ironsrc.com <http://www.ironsrc.com> >> > [image: linkedin] <https://www.linkedin.com/company/ironsource> [image: >> > twitter] <https://twitter.com/ironsource> [image: facebook] >> > <https://www.facebook.com/ironSource> [image: googleplus] >> > <https://plus.google.com/+ironsrc> >> > This email (including any attachments) is for the sole use of the >> intended >> > recipient and may contain confidential information which may be >> protected >> > by legal privilege. If you are not the intended recipient, or the >> employee >> > or agent responsible for delivering it to the intended recipient, you >> are >> > hereby notified that any use, dissemination, distribution or copying of >> > this communication and/or its content is strictly prohibited. If you are >> > not the intended recipient, please immediately notify us by reply email >> or >> > by telephone, delete this email and destroy any copies. Thank you. >> > >> > > > -- > > Nitay Kufert > Backend Team Leader > [image: ironSource] <http://www.ironsrc.com> > > email nita...@ironsrc.com > mobile +972-54-5480021 > fax +972-77-5448273 > skype nitay.kufert.ssa > 121 Menachem Begin St., Tel Aviv, Israel > ironsrc.com <http://www.ironsrc.com> > [image: linkedin] <https://www.linkedin.com/company/ironsource> [image: > twitter] <https://twitter.com/ironsource> [image: facebook] > <https://www.facebook.com/ironSource> [image: googleplus] > <https://plus.google.com/+ironsrc> > This email (including any attachments) is for the sole use of the intended > recipient and may contain confidential information which may be protected > by legal privilege. If you are not the intended recipient, or the employee > or agent responsible for delivering it to the intended recipient, you are > hereby notified that any use, dissemination, distribution or copying of > this communication and/or its content is strictly prohibited. If you are > not the intended recipient, please immediately notify us by reply email or > by telephone, delete this email and destroy any copies. Thank you. > -- Nitay Kufert Backend Team Leader [image: ironSource] <http://www.ironsrc.com> email nita...@ironsrc.com mobile +972-54-5480021 fax +972-77-5448273 skype nitay.kufert.ssa 121 Menachem Begin St., Tel Aviv, Israel ironsrc.com <http://www.ironsrc.com> [image: linkedin] <https://www.linkedin.com/company/ironsource> [image: twitter] <https://twitter.com/ironsource> [image: facebook] <https://www.facebook.com/ironSource> [image: googleplus] <https://plus.google.com/+ironsrc> This email (including any attachments) is for the sole use of the intended recipient and may contain confidential information which may be protected by legal privilege. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication and/or its content is strictly prohibited. If you are not the intended recipient, please immediately notify us by reply email or by telephone, delete this email and destroy any copies. Thank you.
