Re: End-to-end lag spikes when closing a large number of panes

2024-03-20 Thread Caio Camatta via user
Function = custom ProcessWindowFunction > > ) > > > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/operators/windows/#keyed-vs-non-keyed-windows > > -A > > > On Wed, Mar 20, 2024 at 8:55 AM Caio Camatta via user < > user@flink.apache.or

End-to-end lag spikes when closing a large number of panes

2024-03-20 Thread Caio Camatta via user
We run a large-scale Flink 1.16 cluster that uses windowed aggregations and we’re seeing lag spikes on window closure. I’m curious if others have encountered similar issues before and if anyone has suggestions for how to tackle this problem (other than simply increasing parallelism). ContextLag def