Wouldn't it be rather simple to add a retention time on "deleted" items ie keys with null value for topics that are compacted?
The retention time would then be set to some "large" time to allow all consumers to understand that a previous k/v is being deleted. 2015-03-02 17:30 GMT+01:00 Ivan Balashov <ibalas...@gmail.com>: > Guozhang, > > I agree, but upon restart the application still needs to init > KV-storage. And even though values are empty, keys will generate > traffic (delaying app startup time). > Besides, the idea of keeping needless data in kafka forever, even keys > only, sounds rather unsettling. > > I guess we could try to reduce the key update count, and adjust > retention of KV topic. > > Thanks, > > 2015-03-02 19:14 GMT+03:00 Guozhang Wang <wangg...@gmail.com>: > > Currently Kafka log compaction does not support removing keys, but as > long > > as you also have log cleaning done at the app level #.keys will not > > increase indefinitely. >
