I believe "withMaxReadTime()" and "withMaxRecords()" are usually used for testing and debugging purposes and discouraged for being used in production pipelines. They are basically pretty straightforward bounded conversions of unbounded sources that may not perform splitting in an optimal way for example.
Thanks, Cham On Fri, Apr 30, 2021 at 11:26 AM Pablo Estrada <pabl...@google.com> wrote: > Are they experimental? I suppose this is a valid use case, right? I am in > favor of adding a warning, but I don't know if I would call them > experimental. > > I suppose a repeated-batch use case may do this repeatedly (though then > users would need to recover the latest offsets for each partition, which I > guess is not possible at the moment?) > > On Thu, Apr 29, 2021 at 4:17 PM Brian Hulette <bhule...@google.com> wrote: > >> Our oldest open P1 issue is BEAM-6466 - "KafkaIO doesn't commit offsets >> while being used as bounded source" [1]. I'm not sure this is an actual >> issue since KafkaIO doesn't seem to officially support this use-case. The >> relevant parameters indicate they are "mainly used for tests and demo >> applications" [2], and BEAM-2185 - "KafkaIO bounded source" [3] is still >> open. >> >> I think we should close out BEAM-6466 by more clearly indicating that >> withMaxReadTime() and withMaxRecords() are experimental, and/or logging a >> warning when they are used. >> >> I'm happy to make such a change, but I wanted to check if there are any >> objections to this first. >> >> Thanks, >> Brian >> >> [1] https://issues.apache.org/jira/browse/BEAM-6466 >> [2] >> https://github.com/apache/beam/blob/3d4db26cfa4ace0a0f2fbb602f422fe30670c35f/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L960 >> [3] https://issues.apache.org/jira/browse/BEAM-2185 >> >
