I have a simple Flink application with a simple keyBy, a SessionWindow, and
I use an AggregateFunction to incrementally aggregate a result, and write
to a Sink.
Some of the requirements involve accumulating lists of fields from the
events (for example, all URLs), so not all the values in the end s
Hi Ori,
this sounds indeed strange. Can you also reproduce this behavior locally
with a faker source? We should definitely add a profiler and see where
the bottleneck lies.
Which Flink version and state backend are you using?
Regards,
Timo
On 20.10.21 16:17, Ori Popowski wrote:
I have a sim
…
Thias
From: Ori Popowski
Sent: Mittwoch, 20. Oktober 2021 16:17
To: user
Subject: Huge backpressure when using AggregateFunction with Session Window
I have a simple Flink application with a simple keyBy, a SessionWindow, and I
use an AggregateFunction to incrementally aggregate a result, and
I didn't try to reproduce it locally since this job reads 14K events per
second.
I am using Flink version 1.12.1 and RocksDB state backend. It also happens
with Flink 1.10.
I tried to profile with JVisualVM and I didn't see any bottleneck. All the
user functions almost didn't take any CPU time.
O
helps …
>
>
>
> I won’t be able to follow up for the next 1 ½ weeks, unless you try to
> meet me on FlinkForward conference …
>
>
>
> Thias
>
>
>
> *From:* Ori Popowski
> *Sent:* Mittwoch, 20. Oktober 2021 16:17
> *To:* user
> *Subject:* Huge backpress
onnerstag, 21. Oktober 2021 15:32
To: Schwalbe Matthias
Cc: user
Subject: Re: Huge backpressure when using AggregateFunction with Session Window
Thanks for taking the time to answer this.
* You're correct that the SimpleAggregator is not used in the job setup. I
didn't copy th
