Hello Ara, I would love to learn if your rebalance issue mentioned in KAFKA-4392 still exists, since I think what you observed maybe a combination of various issues as we discussed offline, which could be related to KIP-62 as well. Let me know and I'd love to help investigate further.
Guozhang On Wed, Nov 16, 2016 at 6:25 PM, Ara Ebrahimi <ara.ebrah...@argyledata.com> wrote: > This indeed was the issue and cp2 did fix it! Thank you! > > Ara. > > > On Nov 16, 2016, at 4:11 PM, Damian Guy <damian....@gmail.com> wrote: > > > > Hi Ara, > > > > Are you running Kafka Streams v0.10.1? If so, there is a resource leak in > > the window store. See: https://github.com/apache/kafka/pull/2122 > > You can either try checking out the Apache Kafka 0.10.1 branch (which has > > the fix) and building it manually, or you can try using the latest > > confluent packaged version of the library, which you can get from > > confluents maven repository: > > > > <repositories> > > <repository> > > <id>confluent</id> > > <url>http://packages.confluent.io/maven/</url> > > </repository></repositories> > > > > <dependency> > > <groupId>org.apache.kafka</groupId> > > <artifactId>kafka-streams</artifactId> > > <version>0.10.1.0-cp2</version></dependency><dependency> > > <groupId>org.apache.kafka</groupId> > > <artifactId>kafka-clients</artifactId> > > <version>0.10.1.0-cp2</version></dependency> > > > > Thanks, > > > > Damian > > > > > > On Wed, 16 Nov 2016 at 14:52 Ara Ebrahimi <ara.ebrah...@argyledata.com> > > wrote: > > > >> Hi, > >> > >> I have a few KTables in my application. Some of them have unlimited > >> windows. If I leave the application to run for a few hours, I see the > java > >> process consume more and more memory, way above the -Xmx limit. I > >> understand this is due to the rocksdb native lib used by kafka streams. > >> What I don’t understand is how I can control how much rocksdb data > should > >> be kept in memory. I know I can use the RocksDBConfigSetter class, but I > >> need more information on why this unbounded memory allocation happens > and > >> how to tune it. > >> > >> Thanks, > >> Ara. > >> > >> > >> > >> ________________________________ > >> > >> This message is for the designated recipient only and may contain > >> privileged, proprietary, or otherwise confidential information. If you > have > >> received it in error, please notify the sender immediately and delete > the > >> original. Any other use of the e-mail by you is prohibited. Thank you in > >> advance for your cooperation. > >> > >> ________________________________ > >> > > > > > > > > ________________________________ > > > > This message is for the designated recipient only and may contain > privileged, proprietary, or otherwise confidential information. If you have > received it in error, please notify the sender immediately and delete the > original. Any other use of the e-mail by you is prohibited. Thank you in > advance for your cooperation. > > > > ________________________________ > > > > > ________________________________ > > This message is for the designated recipient only and may contain > privileged, proprietary, or otherwise confidential information. If you have > received it in error, please notify the sender immediately and delete the > original. Any other use of the e-mail by you is prohibited. Thank you in > advance for your cooperation. > > ________________________________ > -- -- Guozhang
