On Tue, Mar 12, 2019 at 10:13 AM Maximilian Michels <[email protected]> wrote:
> > I think that is what Max's PR does. KafkaIO writes entire list of values > associated with a key in one transaction. So it depends on how Flink runner > bundles <K, Iter<V>> after a GBK. I would think all of the buffered records > would be queued. Here, the key is the shard id. > > We do not change the execution logic in case of stable input. Elements > will still be processed key-wise. > Wouldn't that require the KafkaEOS to support a different processing mode where the elements are committed with @FinishBundle? The runner could then align bundles and checkpointing as needed. I'm now also curious how @RequiresStableInput is supposed to work with portable pipelines? The runner is not able to inspect the ParDo, so this would need to be provided explicitly as part of the executable stage? > >
