Hi Sophie This will be a good change, I have been thinking about proposing something similar or even passing the properties per store. RocksDB should probably know how much memory was reserved but maybe does not expose it. We are limiting it already as you suggested but this is a rather crude tool. Especially in a larger topology with mixed loads par topic it would be helpful to get more insights which store puts a lot of load on memory. Regarding the limiting capability, I think I remember reading that those only affect some parts of the memory and others can still exceed this limit. I‘ll try to look up the difference. Best regards Patrik
> Am 07.06.2019 um 21:03 schrieb Sophie Blee-Goldman <sop...@confluent.io>: > > Hi Patrik, > > As of 2.3 you will be able to use the RocksDBConfigSetter to effectively > bound the total memory used by RocksDB for a single app instance. You > should already be able to limit the memory used per rocksdb store, though > as you mention there can be a lot of them. I'm not sure you can monitor the > memory usage if you are not limiting it though. > >> On Fri, Jun 7, 2019 at 2:06 AM Patrik Kleindl <pklei...@gmail.com> wrote: >> >> Hi >> Thanks Bruno for the KIP, this is a very good idea. >> >> I have one question, are there metrics available for the memory consumption >> of RocksDB? >> As they are running outside the JVM we have run into issues because they >> were using all the other memory. >> And with multiple streams applications on the same machine, each with >> several KTables and 10+ partitions per topic the number of stores can get >> out of hand pretty easily. >> Or did I miss something obvious how those can be monitored better? >> >> best regards >> >> Patrik >> >>> On Fri, 17 May 2019 at 23:54, Bruno Cadonna <br...@confluent.io> wrote: >>> >>> Hi all, >>> >>> this KIP describes the extension of the Kafka Streams' metrics to include >>> RocksDB's internal statistics. >>> >>> Please have a look at it and let me know what you think. Since I am not a >>> RocksDB expert, I am thankful for any additional pair of eyes that >>> evaluates this KIP. >>> >>> >>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-471:+Expose+RocksDB+Metrics+in+Kafka+Streams >>> >>> Best regards, >>> Bruno >>> >>
