Oh ok, this is a snippet of the code we use :
List<KeyValue<K, V>> keyValues = new ArrayList<>();
try (KeyValueIterator<K, V> iterator = kvStore.all()) {
iterator.forEachRemaining(keyValues::add);
}
// process all entries at once
try {
// business stuff
} catch (Exception e) {
// handling problem
} finally {
// delete them
keyValues.stream().map(item ->
item.key).forEach(kvStore::delete);
}
Something missing ?
On Sat, Feb 4, 2017 at 7:59 PM, Damian Guy <damian....@gmail.com> wrote:
> Keeping the rocksdb iterator wouldn't cause a memory leak in the heap. That
> is why i asked.
>
> On Sat, 4 Feb 2017 at 16:36 Pierre Coquentin <pierre.coquen...@gmail.com>
> wrote:
>
> > The iterator is inside a try-with-resources. And if the memory leak was
> > inside our code, we will see it using visualvm or jmap, and that's not
> the
> > case. This is not a memory leak in the heap. That's why my guess goes
> > directly to rocksdb.
> >
> > On Sat, Feb 4, 2017 at 5:31 PM, Damian Guy <damian....@gmail.com> wrote:
> >
> > > Hi Pierre,
> > >
> > > When you are iterating over the entries do you close the iterator once
> > you
> > > are finished? If you don't then that will cause a memory leak.
> > >
> > > Thanks,
> > > Damian
> > >
> > > On Sat, 4 Feb 2017 at 16:18 Pierre Coquentin <
> pierre.coquen...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We ran a few tests with apache kafka 0.10.1.1.
> > > > We use a Topology with only one processor and a KVStore configured as
> > > > persistent backed by rocksdb 4.9.0. Each events received are stored
> > using
> > > > the method put(key, value) and in the punctuate method, we iterate
> over
> > > all
> > > > entries with all(), processing them and deleting them with
> delete(key).
> > > > Then after a few days, the jvm crashed due to lack of memory.
> Visualvm
> > or
> > > > jmap doesn't show anything, so my guesses were there was a memory
> leak
> > in
> > > > rocksdb. We configured the KVStore to be in memory and as you can see
> > in
> > > > picture 2, the memory is stable.
> > > > I didn't run the same test with a newer version of rocksdb yet.
> > > > Any thoughts ?
> > > > Regards,
> > > >
> > > > Pierre
> > > >
> > >
> >
>
