Hi Lukasz, Thanks for the prompt response. Does it mean that if the side input elements and watermarks got delayed and lagged behind the main input stream, it is considered as an application problem, and Beam runners/engines do not need to handle that?
Best, Shen On Thu, Mar 8, 2018 at 4:15 PM, Lukasz Cwik <lc...@google.com> wrote: > The side input expires relative to the input watermark of the ParDo so > what your suggesting could only happen if the runner had a bug and expired > the side input before it should have happened or the user pipeline has a > bug and is attempting to access a window for something that would always be > considered beyond the maximum lookback which is also an error. > > On Thu, Mar 8, 2018 at 1:07 PM, Shen Li <cs.she...@gmail.com> wrote: > >> Hi, >> >> When a main input element tries to access an expired side input window >> (violating maximumLookback), should ParDo discard the element or treat it >> as an error? >> >> Besides, what should ParDo do in the following situation: >> 1. The side input window W is not expired but unready when the main input >> element X arrives. So the ParDo pushes back the main input element. >> 2. Later the side input window W expires before X is processed. >> In this case, should ParDo throw away X? >> >> Thanks, >> Shen >> > >
