Do you mean if the "startReadTime" as a configuration on its own shifts the
watermark already even if we receive an older entry first? That I do not
know, but based on the code I would say it does not. So if I get it right,
if it would, then my original idea would work fine, and we have to think
fu
These last two emails make sense to me, from both of you.
Does startReadTime impact the watermark in the right way so that late data
is dropped, or whatnot? Because even if you have a partition and you _do_
find an offset that works, late data could come in. So if you restrict the
time to be in th
Well, a sanity check to only allow startReadTime in the past seemed obvious
to me. I can't really think of any common use-case when you want to start
processing after a specific time in the future and only idle until that,
but the other direction has multiple.
The question regarding this IMO is ho
Not a single objection means my idea seems okay then? :)
Balázs Németh ezt írta (időpont: 2022. máj. 26.,
Cs, 5:59):
> https://issues.apache.org/jira/browse/BEAM-14518
>
>
> https://github.com/apache/beam/blob/fd8546355523f67eaddc22249606fdb982fe4938/sdks/java/io/kafka/src/main/java/org/apache/b
https://issues.apache.org/jira/browse/BEAM-14518
https://github.com/apache/beam/blob/fd8546355523f67eaddc22249606fdb982fe4938/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java#L180-L198
Right now the 'startReadTime' config for KafkaIO.Read looks up an offset in
every
