My apologies... After I set the 'maxOffsetsPerTrigger' to a value such as '200000' it started working. Hopefully this will help someone. Thanks.
On Fri, Jun 26, 2020 at 2:12 PM Something Something < mailinglist...@gmail.com> wrote: > My Spark Structured Streaming job works fine when I set "startingOffsets" > to "latest". When I simply change it to "earliest" & specify a new "check > point directory", the job doesn't work. The states don't get timed out > after 10 minutes. > > While debugging I noticed that my 'state' logic is indeed getting executed > but states just don't time out - as they do when I use "latest". Any reason > why? > > Is this a known issue? > > *Note*: I've tried this under Spark 2.3 & 2.4 >
