Hi Gray, Answering your question below: > If we set TOTAL_OFF_HEAP_MEMORY, to, say, 1GB, does that mean the app should only use 1GB for the whole application, or is it per store (so 3GB total)? Or perhaps per store and per task (3GB * num tasks)?
The TOTAL_OFF_HEAP_MEMORY setting limits the memory RocksDB uses *per Kafka Streams client*. That means, if you only use one Kafka Streams client, the TOTAL_OFF_HEAP_MEMORY setting limits the memory for all RocksDB instances. Thank you. Luke On Fri, Nov 19, 2021 at 5:00 AM Gray, John <gray.j...@aoins.com.invalid> wrote: > Hello! We are currently trying our best to limit the amount of memory > RocksDB uses. We have implemented the config setter prescribed in the > RocksDB section of this page: > https://kafka.apache.org/30/documentation/streams/developer-guide/memory-mgmt.html#id3, > but are still hitting OOMs. > > My question is: how is TOTAL_OFF_HEAP_MEMORY used in a streams application > with multiple stores? We specifically have a window store, suppress store, > and an aggregate store we use in a transformer. If we set > TOTAL_OFF_HEAP_MEMORY, to, say, 1GB, does that mean the app should only use > 1GB for the whole application, or is it per store (so 3GB total)? Or > perhaps per store and per task (3GB * num tasks)? > > Thank you for the clarification! >
