Hi Lukasz, Thanks.
> So having the side input significantly delayed can cause a serious backlog on the main input. Sometimes, side input is delayed and it is out of the applications' control. In this situation, should the runner/engine discard pushed back main elements if they access expired side windows, or does Beam expect the runner/engine to terminate the execution? Shen On Thu, Mar 8, 2018 at 4:39 PM, Lukasz Cwik <lc...@google.com> wrote: > The runner/engine is responsible for pushing back the main input until the > side input becomes ready. So having the side input significantly delayed > can cause a serious backlog on the main input. > > On Thu, Mar 8, 2018 at 1:34 PM, Shen Li <cs.she...@gmail.com> wrote: > >> Hi Lukasz, >> >> Thanks for the prompt response. Does it mean that if the side input >> elements and watermarks got delayed and lagged behind the main input >> stream, it is considered as an application problem, and Beam >> runners/engines do not need to handle that? >> >> Best, >> Shen >> >> On Thu, Mar 8, 2018 at 4:15 PM, Lukasz Cwik <lc...@google.com> wrote: >> >>> The side input expires relative to the input watermark of the ParDo so >>> what your suggesting could only happen if the runner had a bug and expired >>> the side input before it should have happened or the user pipeline has a >>> bug and is attempting to access a window for something that would always be >>> considered beyond the maximum lookback which is also an error. >>> >>> On Thu, Mar 8, 2018 at 1:07 PM, Shen Li <cs.she...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> When a main input element tries to access an expired side input window >>>> (violating maximumLookback), should ParDo discard the element or treat it >>>> as an error? >>>> >>>> Besides, what should ParDo do in the following situation: >>>> 1. The side input window W is not expired but unready when the main >>>> input element X arrives. So the ParDo pushes back the main input element. >>>> 2. Later the side input window W expires before X is processed. >>>> In this case, should ParDo throw away X? >>>> >>>> Thanks, >>>> Shen >>>> >>> >>> >> >
