The runner/engine is responsible for pushing back the main input until the side input becomes ready. So having the side input significantly delayed can cause a serious backlog on the main input.
On Thu, Mar 8, 2018 at 1:34 PM, Shen Li <cs.she...@gmail.com> wrote: > Hi Lukasz, > > Thanks for the prompt response. Does it mean that if the side input > elements and watermarks got delayed and lagged behind the main input > stream, it is considered as an application problem, and Beam > runners/engines do not need to handle that? > > Best, > Shen > > On Thu, Mar 8, 2018 at 4:15 PM, Lukasz Cwik <lc...@google.com> wrote: > >> The side input expires relative to the input watermark of the ParDo so >> what your suggesting could only happen if the runner had a bug and expired >> the side input before it should have happened or the user pipeline has a >> bug and is attempting to access a window for something that would always be >> considered beyond the maximum lookback which is also an error. >> >> On Thu, Mar 8, 2018 at 1:07 PM, Shen Li <cs.she...@gmail.com> wrote: >> >>> Hi, >>> >>> When a main input element tries to access an expired side input window >>> (violating maximumLookback), should ParDo discard the element or treat it >>> as an error? >>> >>> Besides, what should ParDo do in the following situation: >>> 1. The side input window W is not expired but unready when the main >>> input element X arrives. So the ParDo pushes back the main input element. >>> 2. Later the side input window W expires before X is processed. >>> In this case, should ParDo throw away X? >>> >>> Thanks, >>> Shen >>> >> >> >
