Hi all, Apologies if this has been asked before, but I could not find the answer to this question. We have a structured streaming job, but for some reason, if we use startingOffsets = latest with foreachbatch mode, it doesn’t produce any data.
Rather, in logs I see it repeats the message “ Fetcher [Consumer] Resetting offset for partition to offset” over and over again.. However with startingOffsets=earliest, we don’t get this issue. I’m wondering then how we can use startingOffsets=latest as I wish to start from the latest offset available. -- Cheers, Ruijing Li
