Hi y'all, I have a reasonably simple KafkaStream application, which merges about 20 topics a few times. The thing is, some of those topic datasets are pretty big, about 10M messages. In total I've got about 200Gb worth of state in RocksDB, the largest topic is 38 Gb.
I had set the MAX_POLL_INTERVAL_MS_CONFIG to one hour to cover the initialization time, but that does not seem nearly enough, I'm looking at more than two hour startup times, and that starts to be a bit ridiculous. Any tips / experiences on how to deal with this case? Move away from Rocks and use an external data store? Any tuning tips on how to tune Rocks to be a bit more useful here? regards, Frank
