Tracked as https://issues.apache.org/jira/browse/BEAM-6879
On Fri, Mar 15, 2019 at 10:13 AM Kenneth Knowles <k...@apache.org> wrote: > Yes, the ParDoPayload has to contain most of the information that is on > DoFnSignature. Everything except the details for feeding the bits to the > Java DoFn. > > Kenn > > On Fri, Mar 15, 2019 at 9:59 AM Reuven Lax <re...@google.com> wrote: > >> I think this attribute needs to be added to the portability protos. >> >> On Fri, Mar 15, 2019 at 9:49 AM Thomas Weise <t...@apache.org> wrote: >> >>> >>> >>> On Tue, Mar 12, 2019 at 10:13 AM Maximilian Michels <m...@apache.org> >>> wrote: >>> >>>> > I think that is what Max's PR does. KafkaIO writes entire list of >>>> values associated with a key in one transaction. So it depends on how Flink >>>> runner bundles <K, Iter<V>> after a GBK. I would think all of the buffered >>>> records would be queued. Here, the key is the shard id. >>>> >>>> We do not change the execution logic in case of stable input. Elements >>>> will still be processed key-wise. >>>> >>> >>> Wouldn't that require the KafkaEOS to support a different processing >>> mode where the elements are committed with @FinishBundle? The runner could >>> then align bundles and checkpointing as needed. >>> >>> I'm now also curious how @RequiresStableInput is supposed to work with >>> portable pipelines? The runner is not able to inspect the ParDo, so this >>> would need to be provided explicitly as part of the executable stage? >>> >>> >>> >>>> >>>>
