Neither, the runner is required to hold back the main input till the side input is ready and also to not expire the side input till the main input watermark advances beyond the garbage collection hold of the side input.
On Thu, Mar 8, 2018 at 1:52 PM, Shen Li <cs.she...@gmail.com> wrote: > Hi Lukasz, > > Thanks. > > > So having the side input significantly delayed can cause a serious > backlog on the main input. > > Sometimes, side input is delayed and it is out of the applications' > control. In this situation, should the runner/engine discard pushed back > main elements if they access expired side windows, or does Beam expect the > runner/engine to terminate the execution? > > > Shen > > > > > On Thu, Mar 8, 2018 at 4:39 PM, Lukasz Cwik <lc...@google.com> wrote: > >> The runner/engine is responsible for pushing back the main input until >> the side input becomes ready. So having the side input significantly >> delayed can cause a serious backlog on the main input. >> >> On Thu, Mar 8, 2018 at 1:34 PM, Shen Li <cs.she...@gmail.com> wrote: >> >>> Hi Lukasz, >>> >>> Thanks for the prompt response. Does it mean that if the side input >>> elements and watermarks got delayed and lagged behind the main input >>> stream, it is considered as an application problem, and Beam >>> runners/engines do not need to handle that? >>> >>> Best, >>> Shen >>> >>> On Thu, Mar 8, 2018 at 4:15 PM, Lukasz Cwik <lc...@google.com> wrote: >>> >>>> The side input expires relative to the input watermark of the ParDo so >>>> what your suggesting could only happen if the runner had a bug and expired >>>> the side input before it should have happened or the user pipeline has a >>>> bug and is attempting to access a window for something that would always be >>>> considered beyond the maximum lookback which is also an error. >>>> >>>> On Thu, Mar 8, 2018 at 1:07 PM, Shen Li <cs.she...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> When a main input element tries to access an expired side input window >>>>> (violating maximumLookback), should ParDo discard the element or treat it >>>>> as an error? >>>>> >>>>> Besides, what should ParDo do in the following situation: >>>>> 1. The side input window W is not expired but unready when the main >>>>> input element X arrives. So the ParDo pushes back the main input element. >>>>> 2. Later the side input window W expires before X is processed. >>>>> In this case, should ParDo throw away X? >>>>> >>>>> Thanks, >>>>> Shen >>>>> >>>> >>>> >>> >> >
