I suppose a production-ready bounded KafkaIO may fetch until reaching the end of each partition(?), or receive a final offset for each partition?
Let's definitely add the warning. Best -P. On Fri, Apr 30, 2021 at 11:33 AM Brian Hulette <bhule...@google.com> wrote: > I guess that is the question. [2] and [3] above make me think that this is > experimental and just not labeled as such. > > It doesn't seem reasonable to have both an open feature request for > bounded KafkaIO (BEAM-2185), and a bug report regarding bounded KafkaIO > (BEAM-6466). > > On Fri, Apr 30, 2021 at 11:26 AM Pablo Estrada <pabl...@google.com> wrote: > >> Are they experimental? I suppose this is a valid use case, right? I am in >> favor of adding a warning, but I don't know if I would call them >> experimental. >> >> I suppose a repeated-batch use case may do this repeatedly (though then >> users would need to recover the latest offsets for each partition, which I >> guess is not possible at the moment?) >> >> On Thu, Apr 29, 2021 at 4:17 PM Brian Hulette <bhule...@google.com> >> wrote: >> >>> Our oldest open P1 issue is BEAM-6466 - "KafkaIO doesn't commit offsets >>> while being used as bounded source" [1]. I'm not sure this is an actual >>> issue since KafkaIO doesn't seem to officially support this use-case. The >>> relevant parameters indicate they are "mainly used for tests and demo >>> applications" [2], and BEAM-2185 - "KafkaIO bounded source" [3] is still >>> open. >>> >>> I think we should close out BEAM-6466 by more clearly indicating that >>> withMaxReadTime() and withMaxRecords() are experimental, and/or logging a >>> warning when they are used. >>> >>> I'm happy to make such a change, but I wanted to check if there are any >>> objections to this first. >>> >>> Thanks, >>> Brian >>> >>> [1] https://issues.apache.org/jira/browse/BEAM-6466 >>> [2] >>> https://github.com/apache/beam/blob/3d4db26cfa4ace0a0f2fbb602f422fe30670c35f/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L960 >>> [3] https://issues.apache.org/jira/browse/BEAM-2185 >>> >>
