Is the idea that we should move away from ValueProviders then? *~Vincent*
On Tue, Mar 23, 2021 at 4:11 PM Robert Bradshaw <[email protected]> wrote: > I would encourage flex templates over further proliferation of > ValueProviders. > > On Tue, Mar 23, 2021 at 12:42 PM Alexey Romanenko < > [email protected]> wrote: > >> I think you are right - now with SDF support in KafkaIO it should be >> possible to determine the number of splits in the runtime and support >> ValueProviders in that way. >> >> CC: Boyuan Zhang >> >> On 23 Mar 2021, at 18:18, Vincent Marquez <[email protected]> >> wrote: >> >> Hello. I was looking at how to use the KafkaIO from a Dataflow Template, >> which requires all configuration options to be ValueProviders, which >> KafkaIO doesn't support currently. I saw this old PR: >> https://github.com/apache/beam/pull/6636/files >> >> I believe the reason this was never merged was there wasn't a good way to >> determine the number of splits to fire up in the UnboundedSource for >> KafkaIO. However, now that KafkaIO is based on SplittableDoFn which >> handles splitting in a dynamic fashion, are there still any blockers to >> this? >> >> Could we change all the KafkaIO parameters to ValueProviders now? >> >> *~Vincent* >> >> >>
