Hi all, During the implementation of this KIP and some offline discussion with Jason Gustafson, I realized now that Kafka is generating a snapshot every hour and the default metadata retention is to delete snapshots after 7 days, every cluster metadata partition will have 168 (1 snapshot per hour * 24 hours per day * 7 days) snapshots. If we assume that in most cases the size of the snapshot is determined by the number of partitions in a cluster, a cluster with 100K partitions will have a snapshot size of roughly 10MB (100 bytes per partition * 100k partitions). For this kind of clusters the cluster metadata partition will always consume around 1.7GB.
We should change the KIP and the default value for `metadata.max.retention.bytes` to 100MB. This should limit the size of the cluster metadata partition for large clusters but keep 7 days worth of snapshots for small clusters. Thanks, -- -José
