-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 If I understand the issue correctly, the problem is that some data, even if it is not updated for a long period of time will, never be deleted because it's always in a segment with new data?
As your segment size is already fairly small, I don't see any other solution as to write corresponding tombstones. Changing the behavior would for sure require a KIP. One could basically change the compaction to leave data that is older than the retention time in their own segments to make them available for deletion. I am not familiar with this part of the code and thus cannot say how complex it would be to implement. - -Matthias On 3/3/20 7:02 PM, Koushik Chitta wrote: > Bubbling this up to understand if anyone else are in similar use > case. > > > -----Original Message----- From: Koushik Chitta > <kchi...@microsoft.com.INVALID> Sent: Sunday, February 23, 2020 > 1:35 PM To: users@kafka.apache.org; d...@kafka.apache.org Subject: > [EXTERNAL] Issue in retention with compact,delete cleanup policy > > Hi, > > I have a Topic with following config. > > cleanup.policy = compact,delete segment.bytes = 52428800 (~52 > mb) min.compaction.lag.ms = 1800000 (30 min) delete.retention.ms = > 86400000 (1 day) retention.ms = 259200000 (3 days) > > Ideally I would want the old records > 3 days to be deleted without > producing an explicit delete(null value of a key) of the record. > But there can be a case due to continuous compaction, the segments > can contain a very old record(eg: > 30 days) and new recent record > (eg: 1hr) which will make the segment ineligible for retention > delete. > > Currently I don't see a work around for this. Please suggest. I > plan to start a KIP to address this use case. > > Thanks, Koushik > -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5fMAEACgkQO4miYXKq /OgBpA//auSjqG8bnpKD44Svey2GK7cI1kA6pyEY/NMgS4PLav5q+dWiPnADBDQV ZmgekEcXLk6TRggl8oHs0zJCf9ETesGwAUQEcQnIstK+lPIBT/fOEpVdJiZzwNt5 24t1flhO3NBgFty+XQm5J0DrNJmMaysbhptuulPpOfn2Cqj6L26Co22rWkm1w9Pa 03bmrswj5f2kVXWvKExY1kZLRxNhGu4fQaovyDF8dgTZAoHQWAwXpUOxB70tl0Ct fDwzzYd+waGNnQ7cbsbdY6QxvhQJj2aijXCgih3tqJ2ww14BIlTHBku9FAIU77Ww OesfAFAt8tcZmYX5pntVyaG2FblATDKSnf1WEAzSRNzuw+dGsnrkAZSwizZJWVYq JgRo3EO55HHPDqol+T1jI/KFL7tUb549rEd6Er3sxM/+PHX7CeKoS+Y/+VOPY8Z3 uc6qVe1y+rjqikBE9dmOjBmJngopbYtAK8Wu6CbdNRqDXXZtkXsnoj1vr7SlsJZ1 yov+11gLEAtGHSrAwGw6W5Jydcwdbxug2xSgWI0W0XD6djiyS3NBw59vBfB4+wPP YLrS5UvxB3W6/wUgivx3iPNTOtHMuqb4lElxVLgHQzKJUmQkouDGQTpUNKCEfncb 9631uJP9hAMLOl/rvPW6CDp2PlMZLAgB14vIgqyEB0vkU7D+1qs= =YTFS -----END PGP SIGNATURE-----
