I would encourage flex templates over further proliferation of ValueProviders.
On Tue, Mar 23, 2021 at 12:42 PM Alexey Romanenko <[email protected]> wrote: > I think you are right - now with SDF support in KafkaIO it should be > possible to determine the number of splits in the runtime and support > ValueProviders in that way. > > CC: Boyuan Zhang > > On 23 Mar 2021, at 18:18, Vincent Marquez <[email protected]> > wrote: > > Hello. I was looking at how to use the KafkaIO from a Dataflow Template, > which requires all configuration options to be ValueProviders, which > KafkaIO doesn't support currently. I saw this old PR: > https://github.com/apache/beam/pull/6636/files > > I believe the reason this was never merged was there wasn't a good way to > determine the number of splits to fire up in the UnboundedSource for > KafkaIO. However, now that KafkaIO is based on SplittableDoFn which > handles splitting in a dynamic fashion, are there still any blockers to > this? > > Could we change all the KafkaIO parameters to ValueProviders now? > > *~Vincent* > > >
