Just added more details on BEAM-6466 <https://issues.apache.org/jira/browse/BEAM-6466>. In short, BEAM-6466 <https://issues.apache.org/jira/browse/BEAM-6466> looks more like a FR instead of a bug to me.
On Fri, Apr 30, 2021 at 12:48 PM Pablo Estrada <[email protected]> wrote: > I suppose a production-ready bounded KafkaIO may fetch until reaching the > end of each partition(?), or receive a final offset for each partition? > > Let's definitely add the warning. > Best > -P. > > On Fri, Apr 30, 2021 at 11:33 AM Brian Hulette <[email protected]> > wrote: > >> I guess that is the question. [2] and [3] above make me think that this >> is experimental and just not labeled as such. >> >> It doesn't seem reasonable to have both an open feature request for >> bounded KafkaIO (BEAM-2185), and a bug report regarding bounded KafkaIO >> (BEAM-6466). >> >> On Fri, Apr 30, 2021 at 11:26 AM Pablo Estrada <[email protected]> >> wrote: >> >>> Are they experimental? I suppose this is a valid use case, right? I am >>> in favor of adding a warning, but I don't know if I would call them >>> experimental. >>> >>> I suppose a repeated-batch use case may do this repeatedly (though then >>> users would need to recover the latest offsets for each partition, which I >>> guess is not possible at the moment?) >>> >>> On Thu, Apr 29, 2021 at 4:17 PM Brian Hulette <[email protected]> >>> wrote: >>> >>>> Our oldest open P1 issue is BEAM-6466 - "KafkaIO doesn't commit offsets >>>> while being used as bounded source" [1]. I'm not sure this is an actual >>>> issue since KafkaIO doesn't seem to officially support this use-case. The >>>> relevant parameters indicate they are "mainly used for tests and demo >>>> applications" [2], and BEAM-2185 - "KafkaIO bounded source" [3] is still >>>> open. >>>> >>>> I think we should close out BEAM-6466 by more clearly indicating that >>>> withMaxReadTime() and withMaxRecords() are experimental, and/or logging a >>>> warning when they are used. >>>> >>>> I'm happy to make such a change, but I wanted to check if there are any >>>> objections to this first. >>>> >>>> Thanks, >>>> Brian >>>> >>>> [1] https://issues.apache.org/jira/browse/BEAM-6466 >>>> [2] >>>> https://github.com/apache/beam/blob/3d4db26cfa4ace0a0f2fbb602f422fe30670c35f/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L960 >>>> [3] https://issues.apache.org/jira/browse/BEAM-2185 >>>> >>>
