Re: Out-of-orderness of window results when testing stateful operators with TextIO

in a specific order. >> >> As I recall on Dataflow runner, there is very limited ordering support. >> Hi +Reuven Lax can share your insights about it? >> >> >> -Rui >> >> >> >> On Sun, Aug 23, 2020 at 8:32 PM Dongwon Kim >> wrote: >&

Out-of-orderness of window results when testing stateful operators with TextIO

Hi, My Beam pipeline is designed to work with an unbounded source KafkaIO. It roughly looks like below: p.apply(KafkaIO.read() ...) // (A-1) .apply(WithKeys.of(...).withKeyType(...)) .apply(Window.into(FixedWindows.of(...))) .apply(Combine.perKey(...))

Re: Support of per-key state after windowing

11:08 AM Kenneth Knowles wrote: > >> >> >> On Sun, Aug 23, 2020 at 1:12 PM Dongwon Kim >> wrote: >> >>> Hi Reuven, >>> >>> You and Kenneth are right; I thought GlobalWindows in unbounded streams >>> need triggers. >>> &g

Re: Support of per-key state after windowing

51 AM Reuven Lax wrote: > You could simply window into GlobalWindows and add a stateful DoFn > afterwards. No need for the triggering and GroupByKey. > > On Sun, Aug 23, 2020 at 9:45 AM Dongwon Kim wrote: > >> Hi Kenneth, >> >> According to your suggestion, I m

Re: Support of per-key state after windowing

on correctly? If not, please advise me with some hints on it. Thanks, Dongwon On Sun, Aug 23, 2020 at 5:10 AM Kenneth Knowles wrote: > Hi Dongwon, > > On Sat, Aug 22, 2020 at 2:46 PM Dongwon Kim wrote: > >> Hi all, >> >> I'm using Beam 2.23.0 with FlinkRunner

Support of per-key state after windowing

Hi all, I'm using Beam 2.23.0 with FlinkRunner and a part of my unbounded pipeline looks like below: > p.apply(WithKeys.of(...).withKeyType(...)) // (A) > .apply(Window.into(FixedWindows.of(...)))// (B) .apply(Combine.perKey(new MyCombinFn())) // (C) .apply(ParDo.of(new