What version do you use? There is a memory leak in the latest version 0.10.1.0. The bug got already fixed in trunk and 0.10.1 branch.
There is already a discussion about a 0.10.1.1 bug fix release. For now, you could build the Kafka Streams from the sources by yourself. -Matthias On 11/29/16 10:30 AM, Jon Yeargers wrote: > My KStreams app seems to be having some memory issues. > > 1. I start it `java -Xmx8G -jar <app>.jar` > > 2. Wait 5-10 minutes - see lots of 'org.apache.zookeeper.ClientCnxn - Got > ping response for sessionid: 0xc58abee3e000013 after 0ms' messages > > 3. When it _finally_ starts reading values it typically goes for a minute > or so, reads a few thousand values and then the OS kills it with 'Out of > memory' error. > > The topology is (essentially): > > stream->aggregate->filter->foreach > > It's reading values and creating a rolling average. > > During phase 2 (above) I see lots of IO wait and the 'scratch' buffer > (usually "/tmp/kafka-streams/appname") fills with 10s of Gb of .. ? (I had > to create a special scratch partition with 100Gb of space as kafka would > fill the / partition and make the system v v unhappy) > > Have I misconfigured something? >
signature.asc
Description: OpenPGP digital signature
