Hi, Thanks for your replies. Yes Terry. You are right. I can try to create a custom source. But perhaps, according to my use case, I figured out I can use a technical field in my data. This is a timestamp and I think I just have to ignore late events with watermarks or later in the pipeline according to metadata stored in the Flink state. I test it now... Thx
David On 2020/01/03 15:44:08, Chesnay Schepler <ches...@apache.org> wrote: > Are you asking how to detect from within the job whether the dump is > complete, or how to combine these 2 jobs? > > If you had a way to notice whether the dump is complete, then I would > suggest to create a custom source that wraps 2 kafka sources, and switch > between them at will based on your conditions. > > > On 03/01/2020 03:53, Terry Wang wrote: > > Hi, > > > > I’d like to share my opinion here. It seems that you need adjust the Kafka > > consumer to have communication each other. When your begin the dump > > process, you need to notify another CDC-topic consumer to wait idle. > > > > > > Best, > > Terry Wang > > > > > > > >> 2020年1月2日 16:49，David Morin <morin.david....@gmail.com> 写道： > >> > >> Hi, > >> > >> Is there a way to stop temporarily to consume one kafka source in > >> streaming mode ? > >> Use case: I have to consume 2 topics but in fact one of them is more > >> prioritized. > >> One of this topic is dedicated to ingest data from db (change data > >> capture) and one of them is dedicated to make a synchronization (a dump > >> i.e. a SELECT ... from db). At the moment the last one is performed by one > >> Flink job and we start this one after stop the previous one (CDC) manually > >> I want to merge these 2 modes and automatically stop consumption of the > >> topic dedicated to the CDC mode when a dump is done. > >> How to handle that with Flink in a streaming way ? backpressure ? ... > >> Thx in advance for your insights > >> > >> David > > > >
