Cool. Are you not using watermark ?
Also, is it possible to start listening offsets from a specific date time ?
Regards
Srini
On Sat, Jun 27, 2020 at 6:12 AM Eric Beabes
wrote:
> My apologies... After I set the 'maxOffsetsPerTrigger' to a value such as
> '20' it started working. Hopefully
My apologies... After I set the 'maxOffsetsPerTrigger' to a value such as
'20' it started working. Hopefully this will help someone. Thanks.
On Fri, Jun 26, 2020 at 2:12 PM Something Something <
mailinglist...@gmail.com> wrote:
> My Spark Structured Streaming job works fine when I set
My Spark Structured Streaming job works fine when I set "startingOffsets"
to "latest". When I simply change it to "earliest" & specify a new "check
point directory", the job doesn't work. The states don't get timed out
after 10 minutes.
While debugging I noticed that my 'state' logic is indeed
