Re: Kafka segment.index.bytes
Thanks, Ismael Updating to Java 8 version 202 solved the problem! On Sun, Feb 24, 2019 at 10:09 AM Soheil Pourbafrani wrote: > Thanks, I use CentOS 7 and Java 8 (192) > > > I'll try updating Java to 202 > > On Sat, Feb 23, 2019 at 11:05 PM Ismael Juma wrote: > >> We use sparse files by default so they don't actually take space (at least >> on Linux). However, there was a bug in Java 8 where sparse files were >> taking all the space. It was fixed in Java 8 update 202. >> >> Ismael >> >> On Sat, Feb 23, 2019 at 2:19 AM Soheil Pourbafrani > > >> wrote: >> >> > Hi, >> > >> > I have a 2-node Kafka cluster. When I create a new topic it preallocates >> > about 21MB for each partition: >> > 10M /data/1-0/.index >> > 0 /data/1-0/.log >> > 10M /data/1-0/.timeindex >> > 4.0K /data/1-0/leader-epoch-checkpoint >> > >> > And because I have many topics with 24 partitions each, about 7GB will >> be >> > preallocated. >> > >> > That behavior is because of property *segment.index.bytes, *but in the >> > document, it is written that We preallocate this index file and shrink >> it >> > only after log rolls. >> > >> > But my question is when exactly Kafka will shrink the preallocated >> space? >> > >> > As I have another standalone Kafka it also preallocates 21MB for each >> > partition but after a few days the indices preallocated space was freed >> and >> > now only 4KB space is allocated, but in 2-nodes Kafka cluster after >> about 2 >> > weeks it's preallocated 7GB for each empty topic. Why? >> > >> >
Re: Kafka segment.index.bytes
Thanks, I use CentOS 7 and Java 8 (192) I'll try updating Java to 202 On Sat, Feb 23, 2019 at 11:05 PM Ismael Juma wrote: > We use sparse files by default so they don't actually take space (at least > on Linux). However, there was a bug in Java 8 where sparse files were > taking all the space. It was fixed in Java 8 update 202. > > Ismael > > On Sat, Feb 23, 2019 at 2:19 AM Soheil Pourbafrani > wrote: > > > Hi, > > > > I have a 2-node Kafka cluster. When I create a new topic it preallocates > > about 21MB for each partition: > > 10M /data/1-0/.index > > 0 /data/1-0/.log > > 10M /data/1-0/.timeindex > > 4.0K /data/1-0/leader-epoch-checkpoint > > > > And because I have many topics with 24 partitions each, about 7GB will be > > preallocated. > > > > That behavior is because of property *segment.index.bytes, *but in the > > document, it is written that We preallocate this index file and shrink it > > only after log rolls. > > > > But my question is when exactly Kafka will shrink the preallocated space? > > > > As I have another standalone Kafka it also preallocates 21MB for each > > partition but after a few days the indices preallocated space was freed > and > > now only 4KB space is allocated, but in 2-nodes Kafka cluster after > about 2 > > weeks it's preallocated 7GB for each empty topic. Why? > > >
Re: Kafka segment.index.bytes
We use sparse files by default so they don't actually take space (at least on Linux). However, there was a bug in Java 8 where sparse files were taking all the space. It was fixed in Java 8 update 202. Ismael On Sat, Feb 23, 2019 at 2:19 AM Soheil Pourbafrani wrote: > Hi, > > I have a 2-node Kafka cluster. When I create a new topic it preallocates > about 21MB for each partition: > 10M /data/1-0/.index > 0 /data/1-0/.log > 10M /data/1-0/.timeindex > 4.0K /data/1-0/leader-epoch-checkpoint > > And because I have many topics with 24 partitions each, about 7GB will be > preallocated. > > That behavior is because of property *segment.index.bytes, *but in the > document, it is written that We preallocate this index file and shrink it > only after log rolls. > > But my question is when exactly Kafka will shrink the preallocated space? > > As I have another standalone Kafka it also preallocates 21MB for each > partition but after a few days the indices preallocated space was freed and > now only 4KB space is allocated, but in 2-nodes Kafka cluster after about 2 > weeks it's preallocated 7GB for each empty topic. Why? >
Kafka segment.index.bytes
Hi, I have a 2-node Kafka cluster. When I create a new topic it preallocates about 21MB for each partition: 10M /data/1-0/.index 0 /data/1-0/.log 10M /data/1-0/.timeindex 4.0K /data/1-0/leader-epoch-checkpoint And because I have many topics with 24 partitions each, about 7GB will be preallocated. That behavior is because of property *segment.index.bytes, *but in the document, it is written that We preallocate this index file and shrink it only after log rolls. But my question is when exactly Kafka will shrink the preallocated space? As I have another standalone Kafka it also preallocates 21MB for each partition but after a few days the indices preallocated space was freed and now only 4KB space is allocated, but in 2-nodes Kafka cluster after about 2 weeks it's preallocated 7GB for each empty topic. Why?
