I think you are right - now with SDF support in KafkaIO it should be possible to determine the number of splits in the runtime and support ValueProviders in that way.
CC: Boyuan Zhang > On 23 Mar 2021, at 18:18, Vincent Marquez <[email protected]> wrote: > > Hello. I was looking at how to use the KafkaIO from a Dataflow Template, > which requires all configuration options to be ValueProviders, which KafkaIO > doesn't support currently. I saw this old PR: > https://github.com/apache/beam/pull/6636/files > <https://github.com/apache/beam/pull/6636/files> > > I believe the reason this was never merged was there wasn't a good way to > determine the number of splits to fire up in the UnboundedSource for KafkaIO. > However, now that KafkaIO is based on SplittableDoFn which handles splitting > in a dynamic fashion, are there still any blockers to this? > > Could we change all the KafkaIO parameters to ValueProviders now? > > ~Vincent
