Flink SQL, temporal joins and backfilling data

2020-12-29 Thread Dan Hill
Hi! I have a Flink SQL job that does a few temporal joins and has been running for over a month on regular data. No issues. Ran well. I'm trying to re-run the Flink SQL job on the same data set but it's failing to checkpoint and very slow to make progress. I've modified some of the checkpoint

Re: Flink SQL, temporal joins and backfilling data

2021-01-17 Thread Dan Hill
Hi Timo. Sorry for the delay. I'll message this message the next time I hit this. I haven't restarted my job in 12 days. I'll check the watermarks the next time I restart. On Tue, Jan 5, 2021 at 4:47 AM Timo Walther wrote: > Hi Dan, > > are you sure that your watermarks are still correct dur

Re: Flink SQL, temporal joins and backfilling data

2021-01-05 Thread Timo Walther
Hi Dan, are you sure that your watermarks are still correct during reprocessing? As far as I know, idle state retention is not used for temporal joins. The watermark indicates when state can be removed in this case. Maybe you can give us some more details about which kind of temporal join yo