Thanks! That thread was really helpful! -Rui
On Wed, Jun 26, 2019 at 1:18 PM Steve Niemitz <sniem...@apache.org> wrote: > There was a thread about this a few months ago as well: > > https://lists.apache.org/thread.html/20d11046d26174969ef44a781e409a1cb9f7c736e605fa40fdf98397@%3Cuser.beam.apache.org%3E > > > On Wed, Jun 26, 2019 at 4:02 PM Robert Bradshaw <rober...@google.com> > wrote: > >> There is no promise that panes will arrive in order (especially the >> further you get "downstream"). Though they may be approximately so, >> it's dangerous to assume that. You can inspect the sequential index in >> PaneInfo to determine whether a pane is older than other panes you >> have seen. >> >> On Wed, Jun 26, 2019 at 7:03 PM Rui Wang <ruw...@google.com> wrote: >> > >> > Hi Community, >> > >> > I am trying to understand Beam model and having a question related to >> accumulating mode and panes: >> > >> > Accumulating mode means every time when a trigger fires, it emits all >> values seen so far in a window(so it's called accumulating), an example >> from Beam programming model guide[1] sets a repeating order has a repeating >> trigger that fires every time 3 elements arrive on a 10-min fixed windowing >> with the following emitted results: >> > >> > First trigger firing: [5, 8, 3] >> > Second trigger firing: [5, 8, 3, 15, 19, 23] >> > Third trigger firing: [5, 8, 3, 15, 19, 23, 9, 13, 10] >> > >> > >> > The original dataflow paper[2] also mentions that accumulating mode is >> useful to downstream consumers to overwrite old value with new value. >> > >> > In order to help such "overwriting" use case, seems to me that Beam >> model provides: >> > 1. triggers fire in order. In the example above, second trigger firing >> should after first trigger firing such that downstream transforms should >> see [5, 8, 3] before [5, 8, 3, 15, 19, 23]. >> > 2. downstream transforms execute panes in order. If this is not true, >> it might end with that new values(from later panes) are overwritten by old >> values(earlier panes) >> > >> > Do I have a correct understanding? >> > >> > >> > Thanks, >> > Rui >> > >> > >> > >> > [1]: >> https://beam.apache.org/documentation/programming-guide/#setting-a-trigger >> > [2]: https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf >> >
