Hi Ashu, others, I have tested with the latest kafkacat with librdkafka 1.2.2 which can also do transactional reading.
Reading the partition with offset reset from beginning will read until offset 10794778 (this is the offset of the LSO that is stuck) Reading the partition from any offset after 10794778 (so any specific offset greater than 10794778, or auto offset reset to latest) will not read anything at all. Reading in uncommitted mode will read properly from any offset. I think my only solution would be to somehow get the LSO on the broker side to increase again. There's nothing I can do on the consumer side to get this working again while keeping read mode read_committed. Best, Pieter ________________________________ Van: Ashutosh singh <getas...@gmail.com> Verzonden: woensdag 20 november 2019 15:15 Aan: Pieter Hameete <pieter.hame...@blockbax.com> CC: users@kafka.apache.org <users@kafka.apache.org> Onderwerp: Re: Last Stable Offset (LSO) stuck for specific topic partition after Broker issues Alright got that. What about resetting or changing the consumer offset ? You can try to change it to some previous offset and restart consumer. Consumer may have to do duplicate processing but should work . On Wed, Nov 20, 2019 at 7:18 PM Pieter Hameete <pieter.hame...@blockbax.com<mailto:pieter.hame...@blockbax.com>> wrote: Hi Ashu, thanks for the tip. We have tried restarting the consumer, but that did not help. All read_committed consumers for this partition (we have multiple) have the same issue. The partition already had different leaders, when we performed a rolling-restart of the brokers. All brokers give the same stuck LSO, so I don't think deleting will the partition will help? It will then restore the partition from another in-sync replica but that also has the incorrect LSO? Best, Pieter ________________________________ Van: Ashutosh singh <getas...@gmail.com<mailto:getas...@gmail.com>> Verzonden: woensdag 20 november 2019 14:43 Aan: users@kafka.apache.org<mailto:users@kafka.apache.org> <users@kafka.apache.org<mailto:users@kafka.apache.org>> Onderwerp: Re: Last Stable Offset (LSO) stuck for specific topic partition after Broker issues Hello Pieter, We had similar issue. Did you try restarting your consumer ? It that doesn't fix then you can try deleting that particular topic partition from the broker and restart the broker so that it will get in sync. Please make sure that you have replica in-sync before deleting the partition. Thanks Ashu On Wed, Nov 20, 2019 at 6:57 PM Pieter Hameete <pieter.hame...@blockbax.com<mailto:pieter.hame...@blockbax.com>> wrote: > Hello, > > after having some Broker issues (too many open files) we managed to > recover our Brokers, but read_committed consumers are stuck for a specific > topic partition. It seems like the LSO is stuck at a specific offset. The > transactional producer for the topic partition is working without errors so > the latest offset is incrementing correctly and so is transactional > producing. > > What could be wrong here? And how can we get this specific LSO to be > increment again? > > Thank you in advance for any advice. > > Best, > > Pieter > -- Thanx & Regard Ashutosh Singh 08151945559 -- Thanx & Regard Ashutosh Singh 08151945559
