I see. For windowed aggregations the disk space (i.e.
"/tmp/kafka-streams/appname") as well as memory consumption on RocksDB
should not keep increasing forever. One thing to note is that you are using
a tumbling window where a new window will be created every minute, so
within 20 minutes of "event
My apologies. In fact the 'aggregate' step includes this: 'TimeWindows.of(20
* 60 * 1000L).advanceBy(60 * 1000L)'
On Tue, Nov 29, 2016 at 9:12 PM, Guozhang Wang wrote:
> Where does the "20 minutes" come from? I thought the "aggregate" operator
> in your
>
> stream->aggregate->filter->foreach
>
>
Where does the "20 minutes" come from? I thought the "aggregate" operator
in your
stream->aggregate->filter->foreach
topology is not a windowed aggregation, so the aggregate results will keep
accumulating.
Guozhang
On Tue, Nov 29, 2016 at 8:40 PM, Jon Yeargers
wrote:
> "keep increasing" - w
"keep increasing" - why? It seems (to me) that the aggregates should be 20
minutes long. After that the memory should be released.
Not true?
On Tue, Nov 29, 2016 at 3:53 PM, Guozhang Wang wrote:
> Jon,
>
> Note that in your "aggregate" function, if it is now windowed aggregate
> then the aggreg
Jon,
Note that in your "aggregate" function, if it is now windowed aggregate
then the aggregation results will keep increasing in your local state
stores unless you're pretty sure that the aggregate key space is bounded.
This is not only related to disk space but also memory since the current
defa
App eventually got OOM-killed. Consumed 53G of swap space.
Does it require a different GC? Some extra settings for the java cmd line?
On Tue, Nov 29, 2016 at 12:05 PM, Jon Yeargers
wrote:
> I cloned/built 10.2.0-SNAPSHOT
>
> App hasn't been OOM-killed yet but it's up to 66% mem.
>
> App takes
I cloned/built 10.2.0-SNAPSHOT
App hasn't been OOM-killed yet but it's up to 66% mem.
App takes > 10 min to start now. Needless to say this is problematic.
The 'kafka-streams' scratch space has consumed 37G and still climbing.
On Tue, Nov 29, 2016 at 10:48 AM, Jon Yeargers
wrote:
> Does eve
Does every broker need to be updated or just my client app(s)?
On Tue, Nov 29, 2016 at 10:46 AM, Matthias J. Sax
wrote:
> What version do you use?
>
> There is a memory leak in the latest version 0.10.1.0. The bug got
> already fixed in trunk and 0.10.1 branch.
>
> There is already a discussion
What version do you use?
There is a memory leak in the latest version 0.10.1.0. The bug got
already fixed in trunk and 0.10.1 branch.
There is already a discussion about a 0.10.1.1 bug fix release. For now,
you could build the Kafka Streams from the sources by yourself.
-Matthias
On 11/29/16 1
