I cloned/built 10.2.0-SNAPSHOT

App hasn't been OOM-killed yet but it's up to 66% mem.

App takes > 10 min to start now. Needless to say this is problematic.

The 'kafka-streams' scratch space has consumed 37G and still climbing.


On Tue, Nov 29, 2016 at 10:48 AM, Jon Yeargers <jon.yearg...@cedexis.com>
wrote:

> Does every broker need to be updated or just my client app(s)?
>
> On Tue, Nov 29, 2016 at 10:46 AM, Matthias J. Sax <matth...@confluent.io>
> wrote:
>
>> What version do you use?
>>
>> There is a memory leak in the latest version 0.10.1.0. The bug got
>> already fixed in trunk and 0.10.1 branch.
>>
>> There is already a discussion about a 0.10.1.1 bug fix release. For now,
>> you could build the Kafka Streams from the sources by yourself.
>>
>> -Matthias
>>
>>
>> On 11/29/16 10:30 AM, Jon Yeargers wrote:
>> > My KStreams app seems to be having some memory issues.
>> >
>> > 1. I start it `java -Xmx8G -jar <app>.jar`
>> >
>> > 2. Wait 5-10 minutes - see lots of 'org.apache.zookeeper.ClientCnxn -
>> Got
>> > ping response for sessionid: 0xc58abee3e000013 after 0ms'  messages
>> >
>> > 3. When it _finally_ starts reading values it typically goes for a
>> minute
>> > or so, reads a few thousand values and then the OS kills it with 'Out of
>> > memory' error.
>> >
>> > The topology is (essentially):
>> >
>> > stream->aggregate->filter->foreach
>> >
>> > It's reading values and creating a rolling average.
>> >
>> > During phase 2 (above) I see lots of IO wait and the 'scratch' buffer
>> > (usually "/tmp/kafka-streams/appname") fills with 10s of Gb of .. ? (I
>> had
>> > to create a special scratch partition with 100Gb of space as kafka would
>> > fill the / partition and make the system v v unhappy)
>> >
>> > Have I misconfigured something?
>> >
>>
>>
>

Reply via email to