I think this attribute needs to be added to the portability protos. On Fri, Mar 15, 2019 at 9:49 AM Thomas Weise <[email protected]> wrote:
> > > On Tue, Mar 12, 2019 at 10:13 AM Maximilian Michels <[email protected]> > wrote: > >> > I think that is what Max's PR does. KafkaIO writes entire list of >> values associated with a key in one transaction. So it depends on how Flink >> runner bundles <K, Iter<V>> after a GBK. I would think all of the buffered >> records would be queued. Here, the key is the shard id. >> >> We do not change the execution logic in case of stable input. Elements >> will still be processed key-wise. >> > > Wouldn't that require the KafkaEOS to support a different processing mode > where the elements are committed with @FinishBundle? The runner could then > align bundles and checkpointing as needed. > > I'm now also curious how @RequiresStableInput is supposed to work with > portable pipelines? The runner is not able to inspect the ParDo, so this > would need to be provided explicitly as part of the executable stage? > > > >> >>
