Mahendra, The WAL is turned off in KafkaStreams. This file is just the rocksdb log, you can probably just delete the old ones: https://github.com/facebook/rocksdb/issues/849 In 0.10.0.1 there is no way of configuring RocksDB via KafkaStreams.
Thanks, Damian On Mon, 20 Mar 2017 at 09:22 Mahendra Kariya <mahendra.kar...@go-jek.com> wrote: > We did some more analysis on why the disk utilisation is continuously > increasing. Turns out it's the RocksDB WAL that's utilising most of the > disk space. The LOG.old WAL files are not getting deleted. Ideally they > should have been. RocksDB provides certain configuration for purging WAL > files > < > https://github.com/facebook/rocksdb/wiki/basic-operations#purging-wal-files > >. > But I am not sure how to set these configs. Any help would be really > appreciated. Just for reference, our Kafka brokers are on v0.10.0.1 and > RocksDB version is 4.8.0. > > > > > On Mon, Mar 20, 2017 at 12:29 PM, Mahendra Kariya < > mahendra.kar...@go-jek.com> wrote: > > > Hey Guozhang, > > > > Thanks a lot for these insights. We are facing the exact same problem as > > Tianji. Our commit frequency is also quite high. We flush almost around > 16K > > messages per minute to Kafka at the end of the topology. > > > > Another issue that we are facing is that rocksdb is not deleting old > data. > > We have set the time window retention duration to 1 hour, but the disk > size > > is constantly increasing. Ideally, the disk utilisation graph should > > plateau after some time. > > > > We would like to apply the config change suggestions that you have given. > > But we are on Kafka 0.10.0.1. And from the docs, it seems > > rocksdb.config.setter is not available for this version. Is there any > > other way for us to configure rocksdb? > > > > Does Kafka 0.10.0.1 emit any rocksdb related metrics that we can monitor > > and set up alerts on? > > > > > > Thanks! > > > > > > >
