Are they experimental? I suppose this is a valid use case, right? I am in favor of adding a warning, but I don't know if I would call them experimental.
I suppose a repeated-batch use case may do this repeatedly (though then users would need to recover the latest offsets for each partition, which I guess is not possible at the moment?) On Thu, Apr 29, 2021 at 4:17 PM Brian Hulette <[email protected]> wrote: > Our oldest open P1 issue is BEAM-6466 - "KafkaIO doesn't commit offsets > while being used as bounded source" [1]. I'm not sure this is an actual > issue since KafkaIO doesn't seem to officially support this use-case. The > relevant parameters indicate they are "mainly used for tests and demo > applications" [2], and BEAM-2185 - "KafkaIO bounded source" [3] is still > open. > > I think we should close out BEAM-6466 by more clearly indicating that > withMaxReadTime() and withMaxRecords() are experimental, and/or logging a > warning when they are used. > > I'm happy to make such a change, but I wanted to check if there are any > objections to this first. > > Thanks, > Brian > > [1] https://issues.apache.org/jira/browse/BEAM-6466 > [2] > https://github.com/apache/beam/blob/3d4db26cfa4ace0a0f2fbb602f422fe30670c35f/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L960 > [3] https://issues.apache.org/jira/browse/BEAM-2185 >
