Hi Is there a way to have a Kafka Connect connector begin in state 'PAUSED'? I.e I would like to have the connector set to paused before it can process any data from Kafka.
Some background: I have a use case where we will push data from Kafka into S3 using Kafka Connect. It also involves a one-time backfill of data from Hadoop to get all the historic data into S3 as well, into the same dataset. To avoid too many duplicates we want the Kafka Connect pipeline and the HDFS-to-S3 pipeline to overlap just a few hours, i.e: 1. start kafka-connect kafka-to-s3 pipeline 2. wait a few hours 3. start pushing data from Hadoop to S3 However, I have one process that deploys Kafka Connect connectors and another one that will handle this backfilling process, so one way of solving this would be if a connector could start in paused state and be resumed by the backfilling process. One less pretty solution to make the connector be paused before it can consume any data is by deploying it with faulty Kafka settings, set it to paused and then correct the settings, but I hope there are better solutions than that regards Rickard -- *Rickard Cardell* Software developer Data Infrastructure Klarna Bank AB (publ) Sveavägen 46, 111 34 Stockholm Tel: +46 8 120 120 00 <+46812012000> Reg no: 556737-0431 klarna.com
