Hi Sachin, We have 6 consumers per node, each managing multiple partitions. We see the memory growing at the start of the application. To get the memory snapshot, download jemalloc here https://github.com/jemalloc/jemalloc/releases, compile and install it using standard command with the option for enabling the profiling ./configure --enable-prof make sudo make install Edit the launcher of your jvm by adding: export LD_PRELOAD=/usr/local/lib/libjemalloc.so export MALLOC_CONF=prof:true,lg_prof_interval:32,prof_prefix:/path/to/store/jeprof 32 means that each 2^32 bytes allocated (4GB), jemalloc will output a file .heap. You will need dot binary to transform the output of jemalloc into an image. It's included in graphviz package. Then you start your jvm and wait. To transform the file, the application need to be running else you won't have the name, only the address. The command is: jeprof --show_bytes --gif /usr/bin/java /path/to/store/jeprof-pid.x.ix.heap > output.gif
On Fri, Feb 10, 2017 at 11:55 AM, Sachin Mittal <sjmit...@gmail.com> wrote: > Hi, > We also seem to be facing some potential rocksdb issue when there are more > than one partition state store created on a machine. > It looks like rocksdb is spending too much time in disk i/o. > > Could you please tell under what case you get the issue and also after how > long running the streams application you face these issues. > > In another application where we run only single partition stream, so only > one state gets created we have never faced any issue and now (same) streams > application is running uninterrupted for months. There we see no cpu > waiting time and all looks to be running fine. > > Also the images you have created, which tool you used to get the jvm memory > snapshot. > > Thanks > Sachin > > > On Fri, Feb 10, 2017 at 1:25 PM, Pierre Coquentin < > pierre.coquen...@gmail.com> wrote: > > > Here is the gist with the two gif > > > > https://gist.github.com/PierreCoquentin/d2df46e5e1c0d3506f6311b343e6f775 > > > > On Fri, Feb 10, 2017 at 7:45 AM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > Pierre, > > > > > > Apache mailing list has some restricts to attach large attachments and > I > > > think that is why your gif files are not shown up. Could you try using > a > > > gist link? > > > > > > Guozhang > > > > > > On Wed, Feb 8, 2017 at 9:49 AM, Pierre Coquentin < > > > pierre.coquen...@gmail.com > > > > wrote: > > > > > > > Well, I am a little perplexed now... I have already recompiled the > > branch > > > > 0.10.1 with rocksdb 4.11.2 and it doesn't seem better. > > > > So I have modified the launcher of our jvm to use jemalloc with the > > > > profile enabled and from the first result I have, it seems that the > > > problem > > > > comes from the method all()... > > > > But I don't understand how the iterator could not be closed correctly > > > with > > > > a try-with-resources... Tomorrow, I will try by calling explicitly > the > > > > method close... > > > > Attached a gif taken after a reboot of the jvm with the profiling > > > enabled, > > > > the second, 12 hours later, we clearly see a problem with the > iterator > > > and > > > > the call of DBIter.SeekToFirst and DBIter.Next which both are called > in > > > the > > > > method RocksDBStore.all(). > > > > > > > > > > > > On Wed, Feb 8, 2017 at 6:30 PM, Guozhang Wang <wangg...@gmail.com> > > > wrote: > > > > > > > >> Hello Pierre, > > > >> > > > >> As Damian said your code looks fine and I cannot think of a direct > > > reason > > > >> for the rocksdb memory leak on top of my head. > > > >> > > > >> Could you build and try out the latest Kafka trunk (will be released > > as > > > >> 0.10.2 in a few days) which contains a newer version of RocksDB and > > see > > > if > > > >> this issue still exists? > > > >> > > > >> > > > >> Guozhang > > > >> > > > >> On Sat, Feb 4, 2017 at 11:54 AM, Damian Guy <damian....@gmail.com> > > > wrote: > > > >> > > > >> > Looks fine > > > >> > > > > >> > On Sat, 4 Feb 2017 at 19:27, Pierre Coquentin < > > > >> pierre.coquen...@gmail.com> > > > >> > wrote: > > > >> > > > > >> > Oh ok, this is a snippet of the code we use : > > > >> > > > > >> > List<KeyValue<K, V>> keyValues = new ArrayList<>(); > > > >> > try (KeyValueIterator<K, V> iterator = kvStore.all()) > { > > > >> > iterator.forEachRemaining(keyValues::add); > > > >> > } > > > >> > // process all entries at once > > > >> > try { > > > >> > // business stuff > > > >> > } catch (Exception e) { > > > >> > // handling problem > > > >> > } finally { > > > >> > // delete them > > > >> > keyValues.stream().map(item -> > > > >> > item.key).forEach(kvStore::delete); > > > >> > } > > > >> > > > > >> > Something missing ? > > > >> > > > > >> > On Sat, Feb 4, 2017 at 7:59 PM, Damian Guy <damian....@gmail.com> > > > >> wrote: > > > >> > > > > >> > > Keeping the rocksdb iterator wouldn't cause a memory leak in the > > > heap. > > > >> > That > > > >> > > is why i asked. > > > >> > > > > > >> > > On Sat, 4 Feb 2017 at 16:36 Pierre Coquentin < > > > >> pierre.coquen...@gmail.com > > > >> > > > > > >> > > wrote: > > > >> > > > > > >> > > > The iterator is inside a try-with-resources. And if the memory > > > leak > > > >> was > > > >> > > > inside our code, we will see it using visualvm or jmap, and > > that's > > > >> not > > > >> > > the > > > >> > > > case. This is not a memory leak in the heap. That's why my > guess > > > >> goes > > > >> > > > directly to rocksdb. > > > >> > > > > > > >> > > > On Sat, Feb 4, 2017 at 5:31 PM, Damian Guy < > > damian....@gmail.com> > > > >> > wrote: > > > >> > > > > > > >> > > > > Hi Pierre, > > > >> > > > > > > > >> > > > > When you are iterating over the entries do you close the > > > iterator > > > >> > once > > > >> > > > you > > > >> > > > > are finished? If you don't then that will cause a memory > leak. > > > >> > > > > > > > >> > > > > Thanks, > > > >> > > > > Damian > > > >> > > > > > > > >> > > > > On Sat, 4 Feb 2017 at 16:18 Pierre Coquentin < > > > >> > > pierre.coquen...@gmail.com > > > >> > > > > > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > Hi, > > > >> > > > > > > > > >> > > > > > We ran a few tests with apache kafka 0.10.1.1. > > > >> > > > > > We use a Topology with only one processor and a KVStore > > > >> configured > > > >> > as > > > >> > > > > > persistent backed by rocksdb 4.9.0. Each events received > are > > > >> stored > > > >> > > > using > > > >> > > > > > the method put(key, value) and in the punctuate method, we > > > >> iterate > > > >> > > over > > > >> > > > > all > > > >> > > > > > entries with all(), processing them and deleting them with > > > >> > > delete(key). > > > >> > > > > > Then after a few days, the jvm crashed due to lack of > > memory. > > > >> > > Visualvm > > > >> > > > or > > > >> > > > > > jmap doesn't show anything, so my guesses were there was a > > > >> memory > > > >> > > leak > > > >> > > > in > > > >> > > > > > rocksdb. We configured the KVStore to be in memory and as > > you > > > >> can > > > >> > see > > > >> > > > in > > > >> > > > > > picture 2, the memory is stable. > > > >> > > > > > I didn't run the same test with a newer version of rocksdb > > > yet. > > > >> > > > > > Any thoughts ? > > > >> > > > > > Regards, > > > >> > > > > > > > > >> > > > > > Pierre > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> -- Guozhang > > > >> > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > >
