Guozhang, Thanks for the suggestion, however, I'm afraid cardinality of keys will grow indefinitely and AFAIU keys are permanent with log compaction. Any chance keys could also be removed during compaction?
Thanks, 2015-03-02 5:27 GMT+03:00 Guozhang Wang <wangg...@gmail.com>: > > From your description it seems Kafka stores "source of truth" of the data > and the k-v store is constructed via consuming from Kafka, right? In that > case time/size-based data retention policy is usually not preferred as it > may delete data out of expectation while people are querying the k-v store. > If you have to enforce some retention policy, then I would suggest use log > compaction at the Kafka layer and use an app-level thread that cleans up > the data in both kafka / kv stored according to your policy.
