Hi Kenn, I just want to confirm that I understand it correctly.
> - You know that W is expired only when you can be sure that no main input element could reference it. This is determined by the *main input* watermark, allowedLateness, and maximumLookback, right? https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/WindowMappingFn.java#L68 > when W expires on the side input you make it ready, you process the elements with empty contents on the side input, then you GC the side input. Even if W is unready according to *side input* watermark, the runner/engine should still make it ready when it violates maximumLookback and *main input* watermark. Is that correct? Thanks, Shen On Thu, Mar 8, 2018 at 9:31 PM, Shen Li <cs.she...@gmail.com> wrote: > I see. Thank you Kenn and Lukasz. > > Best, > Shen > > > On Thu, Mar 8, 2018 at 7:46 PM, Kenneth Knowles <k...@google.com> wrote: > >> I think the description of when a side input is ready vs expired is the >> trouble here. >> >> - You know that W is expired only when you can be sure that no main >> input element could reference it. >> - You know that W is ready *even if it got no data* if the input that >> would end up in W would be dropped (aka when W expires according to the >> *side input* watermark) >> >> So for your scenario, you push back the elements, that holds W from being >> collected, when W expires on the side input you make it ready, you process >> the elements with empty contents on the side input, then you GC the side >> input. >> >> Kenn >> >> On Thu, Mar 8, 2018 at 4:32 PM Shen Li <cs.she...@gmail.com> wrote: >> >>> Hi Lukasz, >>> >>> Let's explain this problem using a specific example. >>> >>> Say I have a main input element X, which accesses side input window W. >>> When X arrives at a ParDo operator, W is not ready and not expired either. >>> So, in this case, the ParDo should push back X and wait for W to become >>> ready. Say, after two minutes, W is still unready but is expired due to >>> advanced main input watermark. In this situation, how does Beam expect >>> runners/engines to handle the pushed back value X? Discard X or throw an >>> error? >>> >>> Thanks, >>> Shen >>> >>> On Thu, Mar 8, 2018 at 6:35 PM, Lukasz Cwik <lc...@google.com> wrote: >>> >>>> I believe your missing over this point: "and also to not expire the >>>> side input till the main input watermark advances beyond the garbage >>>> collection hold of the side input." >>>> >>>> On Thu, Mar 8, 2018 at 3:33 PM, Shen Li <cs.she...@gmail.com> wrote: >>>> >>>>> Hi Lukasz, >>>>> >>>>> Thanks again. >>>>> >>>>> > the runner is required to hold back the main input till the side >>>>> input is ready >>>>> >>>>> Yes, I understand these requirements. But what if the side input >>>>> expires before it becomes ready? >>>>> >>>>> Shen >>>>> >>>>> >>>> >>> >
