Sound like a workflow/pipeline thing in jenkins (or equivalent) to me.
Den ons 26 sep. 2018 kl 17:27 skrev Rickard Cardell <rickard.card...@klarna.com.invalid>: > Hi > Is there a way to have a Kafka Connect connector begin in state 'PAUSED'? > I.e I would like to have the connector set to paused before it can process > any data from Kafka. > > Some background: > > I have a use case where we will push data from Kafka into S3 using Kafka > Connect. It also involves a one-time backfill of data from Hadoop to get > all the historic data into S3 as well, into the same dataset. > > To avoid too many duplicates we want the Kafka Connect pipeline and the > HDFS-to-S3 pipeline to overlap just a few hours, i.e: > > 1. start kafka-connect kafka-to-s3 pipeline > 2. wait a few hours > 3. start pushing data from Hadoop to S3 > > However, I have one process that deploys Kafka Connect connectors and > another one that will handle this backfilling process, so one way of > solving this would be if a connector could start in paused state and be > resumed by the backfilling process. > > One less pretty solution to make the connector be paused before it can > consume any data is by deploying it with faulty Kafka settings, set it to > paused and then correct the settings, but I hope there are better solutions > than that > > regards > Rickard > > -- > > *Rickard Cardell* > Software developer > Data Infrastructure > > Klarna Bank AB (publ) > Sveavägen 46, 111 34 Stockholm > Tel: +46 8 120 120 00 <+46812012000> > Reg no: 556737-0431 > klarna.com >
