Yes, the ParDoPayload has to contain most of the information that is on DoFnSignature. Everything except the details for feeding the bits to the Java DoFn.
Kenn On Fri, Mar 15, 2019 at 9:59 AM Reuven Lax <[email protected]> wrote: > I think this attribute needs to be added to the portability protos. > > On Fri, Mar 15, 2019 at 9:49 AM Thomas Weise <[email protected]> wrote: > >> >> >> On Tue, Mar 12, 2019 at 10:13 AM Maximilian Michels <[email protected]> >> wrote: >> >>> > I think that is what Max's PR does. KafkaIO writes entire list of >>> values associated with a key in one transaction. So it depends on how Flink >>> runner bundles <K, Iter<V>> after a GBK. I would think all of the buffered >>> records would be queued. Here, the key is the shard id. >>> >>> We do not change the execution logic in case of stable input. Elements >>> will still be processed key-wise. >>> >> >> Wouldn't that require the KafkaEOS to support a different processing mode >> where the elements are committed with @FinishBundle? The runner could then >> align bundles and checkpointing as needed. >> >> I'm now also curious how @RequiresStableInput is supposed to work with >> portable pipelines? The runner is not able to inspect the ParDo, so this >> would need to be provided explicitly as part of the executable stage? >> >> >> >>> >>>
