Currently Kafka log compaction does not support removing keys, but as long as you also have log cleaning done at the app level #.keys will not increase indefinitely.
On Mon, Mar 2, 2015 at 2:08 AM, Ivan Balashov <ibalas...@gmail.com> wrote: > Guozhang, > > Thanks for the suggestion, however, I'm afraid cardinality of keys > will grow indefinitely and AFAIU keys are permanent with log > compaction. Any chance keys could also be removed during compaction? > > Thanks, > > 2015-03-02 5:27 GMT+03:00 Guozhang Wang <wangg...@gmail.com>: > > > > From your description it seems Kafka stores "source of truth" of the data > > and the k-v store is constructed via consuming from Kafka, right? In that > > case time/size-based data retention policy is usually not preferred as it > > may delete data out of expectation while people are querying the k-v > store. > > If you have to enforce some retention policy, then I would suggest use > log > > compaction at the Kafka layer and use an app-level thread that cleans up > > the data in both kafka / kv stored according to your policy. > -- -- Guozhang
