Re: Effect of settings segment.ms and retention.ms not accurate

2018-05-29 Thread Shantanu Deshmukh
This is helpful. Thanks a lot :-) On Tue, May 29, 2018 at 11:47 PM Matthias J. Sax wrote: > ConsumerRecord#timestamp() > > similar to ConsumerRecord#key() and ConsumerRecord#value() > > > -Matthias > > On 5/28/18 11:22 PM, Shantanu Deshmukh wrote: > > But then I wonder, why such things are not m

Re: Effect of settings segment.ms and retention.ms not accurate

2018-05-29 Thread Matthias J. Sax
ConsumerRecord#timestamp() similar to ConsumerRecord#key() and ConsumerRecord#value() -Matthias On 5/28/18 11:22 PM, Shantanu Deshmukh wrote: > But then I wonder, why such things are not mentioned anywhere in Kafka > configuration document? I relied on that setting and it caused us some > issue

Re: Effect of settings segment.ms and retention.ms not accurate

2018-05-28 Thread Shantanu Deshmukh
But then I wonder, why such things are not mentioned anywhere in Kafka configuration document? I relied on that setting and it caused us some issues. If it is mentioned clearly then everyone will be aware. Could you please point in right direction about reading timestamp of log message? I will see

Re: Effect of settings segment.ms and retention.ms not accurate

2018-05-28 Thread Matthias J. Sax
Retention time is a lower bound for how long it is guaranteed that data will be stored. This guarantee work "one way" only. There is no guarantee when data will be deleted after the bound passed. However, client side, you can always check the record timestamp and just drop older data that is still

Re: Effect of settings segment.ms and retention.ms not accurate

2018-05-28 Thread Shantanu Deshmukh
Please help. On Mon, May 28, 2018 at 5:18 PM Shantanu Deshmukh wrote: > I have a topic otp-sms. I want that retention of this topic should be 5 > minutes as OTPs are invalid post that amount of time. So I set > retention.ms=30. However, this was not working. So reading more in > depth in Ka

Effect of settings segment.ms and retention.ms not accurate

2018-05-28 Thread Shantanu Deshmukh
I have a topic otp-sms. I want that retention of this topic should be 5 minutes as OTPs are invalid post that amount of time. So I set retention.ms=30. However, this was not working. So reading more in depth in Kafka configuration document I found another topic level setting that can be tuned