In Kafka 2.0, default offsets retention period is changed to 7 days[a] and
better expiration semantics of consumer groups was added as part of
KIP-211[b].
a - https://kafka.apache.org/documentation/#upgrade_200_notable
b -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-211%3A+Revise+Expira
Hi Manikumar,
Thanks for the reply, it seems you caught the correct nerve as we never set
this property, and if kafka is taking the default value then it would mean
that the consumer group would expire in a day, which completely explains
our behaviour.
Thanks a lot. We will test this and verify
What is the value set for offsets.retention.minutes? Default value is
`1day` in Kafka 0.11 release.
Based on your requirement, you may need to increase this value.
On Mon, Aug 27, 2018 at 1:33 PM Biplob Biswas
wrote:
> Hi,
>
> Can anyone provide any insights to the issue we faced?
>
> Thanks &
Hi,
Can anyone provide any insights to the issue we faced?
Thanks & Regards
Biplob Biswas
On Fri, Aug 24, 2018 at 4:50 PM Biplob Biswas
wrote:
> Hi everyone,
>
> *Short Version:*
> We had an unusual situation where after restart of our spark job, rather
> than reading from the last known offs
Hi everyone,
*Short Version:*
We had an unusual situation where after restart of our spark job, rather
than reading from the last known offset of the consumer group, it started
consuming from the latest offset, thus leading to data loss.
*Long Version:*
We have a spark job which crashed last Frid
