On Thu, Oct 24, 2019 at 7:13 AM Eno Thereska <eno.there...@gmail.com> wrote:
> Going back to initial thread with general questions on KIP. I think > aspects of the user experience still need clarification: > > - if a user has a mix of compacted and non-compacted topics it will be > hard to reason about storage needs overall. Could you give a reason > why compacted topics are not supported? This is probably because to do > that you'd have to go with a paging approach (like Ryanne earlier > suggested) and that will be expensive in terms of IO. Do you want to > discount supporting compacted topics this early in the KIP design or > do you want to leave open the option of supporting them eventually? In > an ideal system, Kafka figures out if the topic is compacted or not > and for non-compacted topics it doesn't do the local copy so it goes > through a fast path. > > Hi Eno, I think the main purpose of tiered storage is to save local disk space. Compact topics are just in-memory hash tables. They are anyway very small. They can be totally ignored when you calculate local storage needs. The segment files in remote storage are considered as read-only. They are not going to be changed until deleted after expire. The segment files of a compact topic have to be compacted / rewritten periodically. Doing this on remote storage can may things much more complicated.
