Hi Jason, It is still not clear to me how adding a callback would help. You could remove the periodic task, but you would still need to scan the entire cache to remove the expired offsets. Am i missing something?
Thanks, Damian On 11 August 2016 at 10:00, Damian Guy <damian....@gmail.com> wrote: > Hi Jason, > > Thanks for your input - appreciated. > > 1. Would it make sense to use this KIP in the consumer coordinator to >> expire offsets based on the topic's retention time? Currently, we have a >> periodic task which scans the full cache to check which offsets can be >> expired, but we might be able to get rid of this if we had a callback to >> update the cache when a segment was deleted. Technically offsets can be >> given their own expiration time, but it seems questionable whether we need >> this going forward (the new consumer doesn't even expose it at the >> moment). >> > > The KIP in its current form isn't adding a callback. So you'd still need > to scan the cache and remove any expired offsets, however you wouldn't send > the tombstone messages. > Having a callback sounds useful, though it isn't clear to me how you would > know which offsets to remove from the cache on segment deletion? I will > look into it. > > >> 2. This KIP could also be useful for expiration in the case of a cache >> maintained on the client, but I don't see an obvious way that we'd be able >> to leverage it since there's no indication to the client when a segment >> has >> been deleted (unless they reload the cache from the beginning of the log). >> One approach I can think of would be to write corresponding tombstones as >> necessary when a segment is removed, but that seems pretty heavy. Have you >> considered this problem? >> >> > We've not considered this and I'm not sure we want to as part of this KIP. > > Thanks, > Damian > > >> >> >> On Mon, Aug 8, 2016 at 12:41 AM, Damian Guy <damian....@gmail.com> wrote: >> >> > Hi, >> > >> > We have created KIP 71: Enable log compaction and deletion to co-exist` >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> > 71%3A+Enable+log+compaction+and+deletion+to+co-exist >> > >> > Please take a look. Feedback is appreciated. >> > >> > Thank you >> > >> > >
