[ https://issues.apache.org/jira/browse/KAFKA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Harris updated KAFKA-15470: -------------------------------- Component/s: (was: connect) > Allow creating connectors in a stopped state > -------------------------------------------- > > Key: KAFKA-15470 > URL: https://issues.apache.org/jira/browse/KAFKA-15470 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect > Reporter: Yash Mayya > Assignee: Yash Mayya > Priority: Major > Labels: connect, kafka-connect, kip-required > Fix For: 3.7.0 > > > [KIP-875: First-class offsets support in Kafka > Connect|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect] > introduced a new {{STOPPED}} state for connectors along with some REST API > endpoints to retrieve and modify offsets for connectors. Currently, only > connectors that already exist can be stopped and any newly created connector > will always be in the {{RUNNING}} state initially. Allowing the creation of > connectors in a {{STOPPED}} state will facilitate multiple new use cases. One > interesting use case would be to migrate connectors from one Kafka Connect > cluster to another. Individual connector migration would be useful in a > number of scenarios such as breaking a large cluster into multiple smaller > clusters (or vice versa), moving a connector from a cluster running in one > data center to another etc. A connector migration could be achieved by using > the following sequence of steps :- > # Stop the running connector on the original Kafka Connect cluster > # Retrieve the offsets for the connector via the {{GET > /connectors/\{connector}/offsets}} endpoint > # Create the connector in a stopped state using the same configuration on > the new Kafka Connect cluster > # Alter the offsets for the connector on the new cluster via the {{PATCH > /connectors/\{connector}/offsets}} endpoint (using the offsets obtained from > the original cluster) > # Resume the connector on the new cluster and delete it on the original > cluster > Another use case for creating connectors in a stopped state could be > deploying connectors as a part of a larger data pipeline before the source / > sink data system has been created or is ready for data transfer. -- This message was sent by Atlassian Jira (v8.20.10#820010)