Looks fine On Sat, 4 Feb 2017 at 19:27, Pierre Coquentin <pierre.coquen...@gmail.com> wrote:
Oh ok, this is a snippet of the code we use : List<KeyValue<K, V>> keyValues = new ArrayList<>(); try (KeyValueIterator<K, V> iterator = kvStore.all()) { iterator.forEachRemaining(keyValues::add); } // process all entries at once try { // business stuff } catch (Exception e) { // handling problem } finally { // delete them keyValues.stream().map(item -> item.key).forEach(kvStore::delete); } Something missing ? On Sat, Feb 4, 2017 at 7:59 PM, Damian Guy <damian....@gmail.com> wrote: > Keeping the rocksdb iterator wouldn't cause a memory leak in the heap. That > is why i asked. > > On Sat, 4 Feb 2017 at 16:36 Pierre Coquentin <pierre.coquen...@gmail.com> > wrote: > > > The iterator is inside a try-with-resources. And if the memory leak was > > inside our code, we will see it using visualvm or jmap, and that's not > the > > case. This is not a memory leak in the heap. That's why my guess goes > > directly to rocksdb. > > > > On Sat, Feb 4, 2017 at 5:31 PM, Damian Guy <damian....@gmail.com> wrote: > > > > > Hi Pierre, > > > > > > When you are iterating over the entries do you close the iterator once > > you > > > are finished? If you don't then that will cause a memory leak. > > > > > > Thanks, > > > Damian > > > > > > On Sat, 4 Feb 2017 at 16:18 Pierre Coquentin < > pierre.coquen...@gmail.com > > > > > > wrote: > > > > > > > Hi, > > > > > > > > We ran a few tests with apache kafka 0.10.1.1. > > > > We use a Topology with only one processor and a KVStore configured as > > > > persistent backed by rocksdb 4.9.0. Each events received are stored > > using > > > > the method put(key, value) and in the punctuate method, we iterate > over > > > all > > > > entries with all(), processing them and deleting them with > delete(key). > > > > Then after a few days, the jvm crashed due to lack of memory. > Visualvm > > or > > > > jmap doesn't show anything, so my guesses were there was a memory > leak > > in > > > > rocksdb. We configured the KVStore to be in memory and as you can see > > in > > > > picture 2, the memory is stable. > > > > I didn't run the same test with a newer version of rocksdb yet. > > > > Any thoughts ? > > > > Regards, > > > > > > > > Pierre > > > > > > > > > >