RE: [EXTERNAL] Re: Issue in retention with compact,delete cleanup policy

2020-03-04 Thread Koushik Chitta
Right. Let me start a kip to add in few ideas to discuss . Thanks, Koushik -Original Message- From: Matthias J. Sax Sent: Tuesday, March 3, 2020 8:35 PM To: users@kafka.apache.org Subject: [EXTERNAL] Re: Issue in retention with compact,delete cleanup policy -BEGIN PGP SIGNED

Re: Issue in retention with compact,delete cleanup policy

2020-03-03 Thread Matthias J. Sax
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 If I understand the issue correctly, the problem is that some data, even if it is not updated for a long period of time will, never be deleted because it's always in a segment with new data? As your segment size is already fairly small, I don't see

RE: Issue in retention with compact,delete cleanup policy

2020-03-03 Thread Koushik Chitta
Bubbling this up to understand if anyone else are in similar use case. -Original Message- From: Koushik Chitta Sent: Sunday, February 23, 2020 1:35 PM To: users@kafka.apache.org; d...@kafka.apache.org Subject: [EXTERNAL] Issue in retention with compact,delete cleanup policy Hi, I have