[ https://issues.apache.org/jira/browse/KAFKA-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17756048#comment-17756048 ]
Greg Harris commented on KAFKA-15372: ------------------------------------- [~durban] thank you for the sanity check, I see the flaw now. The NotLeaderException is created/thrown in the Herder, and the caller (typically the ConnectorsResource) is responsible for handling the error and forwarding to the leader. See HerderRequestHandler#completeOrForwardRequest. I thought the MirrorMaker class used this handler for applying the connector configuration on startup, but it doesn't. It never triggers the forwarding logic that normal REST requests have. More importantly, KIP-710 only added {_}internal rest resources{_}, which are for writing task configurations and fencing zombies. It didn't include the connector configuration endpoint, which is typically public-api. This was an oversight in the KIP-710 design/implementation, and should be fixed. Currently, KIP-710 only benefits the internal herder requests for writing task configurations and fencing zombies, it is not capable of forwarding connector configurations. Since KIP-710 already added the necessary configuration options to enable/disable the internal REST, and this arguably should have been covered by the original implementation, I think we can address this in a bug-fix. To workaround this issue, users will need to fully stop and fully start MM2, to allow the first node to apply configurations successfully. > MM2 rolling restart can drop configuration changes silently > ----------------------------------------------------------- > > Key: KAFKA-15372 > URL: https://issues.apache.org/jira/browse/KAFKA-15372 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker > Reporter: Daniel Urban > Priority: Major > > When MM2 is restarted, it tries to update the Connector configuration in all > flows. This is a one-time trial, and fails if the Connect worker is not the > leader of the group. > In a distributed setup and with a rolling restart, it is possible that for a > specific flow, the Connect worker of the just restarted MM2 instance is not > the leader, meaning that Connector configurations can get dropped. > For example, assuming 2 MM2 instances, and one flow A->B: > # MM2 instance 1 is restarted, the worker inside MM2 instance 2 becomes the > leader of A->B Connect group. > # MM2 instance 1 tries to update the Connector configurations, but fails > (instance 2 has the leader, not instance 1) > # MM2 instance 2 is restarted, leadership moves to worker in MM2 instance 1 > # MM2 instance 2 tries to update the Connector configurations, but fails > At this point, the configuration changes before the restart are never > applied. Many times, this can also happen silently, without any indication. -- This message was sent by Atlassian Jira (v8.20.10#820010)