Re: How to advance Kafka offsets manually with checkpointing enabled on Flink (TableAPI)

2021-03-19 Thread Rex Fenley
Thanks for the info! On Thu, Mar 18, 2021 at 7:46 AM Dawid Wysakowicz wrote: > Hi Rex, > > The approach you described is definitely possible in the DataStream API. > You could replace the uid of your Kafka source and start your job with your > checkpoint with the allowNonRestoredState option

Re: How to advance Kafka offsets manually with checkpointing enabled on Flink (TableAPI)

2021-03-18 Thread Dawid Wysakowicz
Hi Rex, The approach you described is definitely possible in the DataStream API. You could replace the uid of your Kafka source and start your job with your checkpoint with the allowNonRestoredState option enabled[1]. I am afraid though it is not possible to change the uid in Table API/SQL

How to advance Kafka offsets manually with checkpointing enabled on Flink (TableAPI)

2021-03-16 Thread Rex Fenley
Hello, I'm wondering how, in the event of a poison pill record on Kafka, to advance a partition's checkpointed offsets by 1 when using the TableAPI/SQL. It is my understanding that when checkpointing is enabled Flink uses its own checkpoint committed offsets and not the offsets committed to