[ https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546263#comment-15546263 ]
Kenneth Knowles edited comment on BEAM-696 at 10/11/16 6:52 PM: ---------------------------------------------------------------- The SDK needs to lay out a spec, which I think is what Pei is saying, and the runner comes up with the execution plan. To clarify - are you suggesting that {{CombineFn}} should only be allowed side input access in {{extractOutput}}, or are you suggesting that runners be required to wait until {{extractOutput}} _will_ be called before running a sequence of {{addInput}}* {{mergeAccum}}* {{extractoutput}} that accesses side inputs? The latter sounds like it could be loosened to "give a consistent view of a side input to the sequence of {{addInput}}* {{mergeAccum}}* {{extractOutput}}" and your proposed execution plan is one obvious choice of how to achieve it. was (Author: kenn): The SDK needs to lay out a spec, which I think is what Pei is saying, and the runner comes up with the execution plan. To clarify - are you suggesting that {{CombineFn}} should only be allowed side input access in {{extractOutput}}, or are you suggesting that runners be required to wait until {{extractOutput}} _will_ be called before running a sequence of {{addInput}}* {{mergeAccum}}* {{extractoutput}} that accesses side inputs? The latter sounds like it could be loosed to "give a consistent view of a side input to the sequence of {{addInput}}* {{mergeAccum}}* {{extractOutput}}" and your proposed execution plan is one obvious choice of how to achieve it. > Side-Inputs non-deterministic with merging main-input windows > ------------------------------------------------------------- > > Key: BEAM-696 > URL: https://issues.apache.org/jira/browse/BEAM-696 > Project: Beam > Issue Type: Bug > Components: beam-model > Reporter: Ben Chambers > Assignee: Pei He > > Side-Inputs are non-deterministic for several reasons: > 1. Because they depend on triggering of the side-input (this is acceptable > because triggers are by their nature non-deterministic). > 2. They depend on the current state of the main-input window in order to > lookup the side-input. This means that with merging > 3. Any runner optimizations that affect when the side-input is looked up may > cause problems with either or both of these. > This issue focuses on #2 -- the non-determinism of side-inputs that execute > within a Merging WindowFn. > Possible solution would be to defer running anything that looks up the > side-input until we need to extract an output, and using the main-window at > that point. Specifically, if the main-window is a MergingWindowFn, don't > execute any kind of pre-combine, instead buffer all the inputs and combine > later. > This could still run into some non-determinism if there are triggers > controlling when we extract output. -- This message was sent by Atlassian JIRA (v6.3.4#6332)