Re: Avoiding deadlock with iterations

2018-01-25 Thread Piotr Nowojski
Hi, This is a known problem and I don’t think there is an easy solution to this. Please refer to the: http://mail-archives.apache.org/mod_mbox/flink-user/201704.mbox/%3c5486a7fd-41c3-4131-5100-272825088...@gaborhermann.com%3E

Avoiding deadlock with iterations

2018-01-24 Thread Ken Krugler
Hi all, We’ve run into deadlocks with two different streaming workflows that have iterations. In both cases, the issue is with fan-out; if any operation in the loop can emit more records than consumed, eventually a network buffer fills up, and then everyone in the iteration loop is blocked.