We use sparse files by default so they don't actually take space (at least on Linux). However, there was a bug in Java 8 where sparse files were taking all the space. It was fixed in Java 8 update 202.
Ismael On Sat, Feb 23, 2019 at 2:19 AM Soheil Pourbafrani <soheil.i...@gmail.com> wrote: > Hi, > > I have a 2-node Kafka cluster. When I create a new topic it preallocates > about 21MB for each partition: > 10M /data/1-0/00000000000000000000.index > 0 /data/1-0/00000000000000000000.log > 10M /data/1-0/00000000000000000000.timeindex > 4.0K /data/1-0/leader-epoch-checkpoint > > And because I have many topics with 24 partitions each, about 7GB will be > preallocated. > > That behavior is because of property *segment.index.bytes, *but in the > document, it is written that We preallocate this index file and shrink it > only after log rolls. > > But my question is when exactly Kafka will shrink the preallocated space? > > As I have another standalone Kafka it also preallocates 21MB for each > partition but after a few days the indices preallocated space was freed and > now only 4KB space is allocated, but in 2-nodes Kafka cluster after about 2 > weeks it's preallocated 7GB for each empty topic. Why? >
