Hi Frank, Is this on a restart of the application?
Thanks, Damian On Fri, 25 Nov 2016 at 11:09 Frank Lyaruu <flya...@gmail.com> wrote: > Hi y'all, > > I have a reasonably simple KafkaStream application, which merges about 20 > topics a few times. > The thing is, some of those topic datasets are pretty big, about 10M > messages. In total I've got > about 200Gb worth of state in RocksDB, the largest topic is 38 Gb. > > I had set the MAX_POLL_INTERVAL_MS_CONFIG to one hour to cover the > initialization time, > but that does not seem nearly enough, I'm looking at more than two hour > startup times, and > that starts to be a bit ridiculous. > > Any tips / experiences on how to deal with this case? Move away from Rocks > and use an external > data store? Any tuning tips on how to tune Rocks to be a bit more useful > here? > > regards, Frank >
