Thanks Timo for suggested solution. Will go with idea of artificial key for
our use case.
Gagan
On Mon, Jan 7, 2019 at 10:21 PM Timo Walther wrote:
> Hi Gagan,
>
> a typical solution to such a problem is to introduce an artifical key
> (enrichment id + some additional suffix), you can then keyB
Hi Gagan,
a typical solution to such a problem is to introduce an artifical key
(enrichment id + some additional suffix), you can then keyBy on this
artificial key and thus spread the workload more evenly. Of course you
need to make sure that records of the second stream are duplicated to
all
Flink Version is 1.7.
Thanks Zhijiang for your pointer. Initially I was checking only for few.
However I just checked for all and found couple of them having queue length
of 40+ which seems to be due to skewness in data. Is there any general
guide lines on how to handle skewed data? In my case I am
Hi Gagan,
What flink version do you use? And have you checked the
buffers.inputQueueLength for all the related parallelism (connected with A) of
B? It may exist the scenario that only one parallelim B is full of inqueue
buffers which back pressure A, and the input queue for other parallelism B
Hi,
I want to understand does any of buffer stats help in debugging /
validating that downstream operator is performing slow when Back Pressure
is high? Say I have A -> B operators and A shows High Back Pressure which
indicates something wrong or not performing well on B side which is slowing
down
