@Damian: Yes, it ran before, and it has that 200gb blob worth of Rocksdb stuff
@Svente: It's on a pretty high end san in a managed private cloud, I'm unsure what the ultimate storage is, but I doubt there is a performance problem there. On Fri, 25 Nov 2016 at 13:37, Svante Karlsson <svante.karls...@csi.se> wrote: > What kind of disk are you using for the rocksdb store? ie spinning or ssd? > > 2016-11-25 12:51 GMT+01:00 Damian Guy <damian....@gmail.com>: > > > Hi Frank, > > > > Is this on a restart of the application? > > > > Thanks, > > Damian > > > > On Fri, 25 Nov 2016 at 11:09 Frank Lyaruu <flya...@gmail.com> wrote: > > > > > Hi y'all, > > > > > > I have a reasonably simple KafkaStream application, which merges about > 20 > > > topics a few times. > > > The thing is, some of those topic datasets are pretty big, about 10M > > > messages. In total I've got > > > about 200Gb worth of state in RocksDB, the largest topic is 38 Gb. > > > > > > I had set the MAX_POLL_INTERVAL_MS_CONFIG to one hour to cover the > > > initialization time, > > > but that does not seem nearly enough, I'm looking at more than two hour > > > startup times, and > > > that starts to be a bit ridiculous. > > > > > > Any tips / experiences on how to deal with this case? Move away from > > Rocks > > > and use an external > > > data store? Any tuning tips on how to tune Rocks to be a bit more > useful > > > here? > > > > > > regards, Frank > > > > > >
