Jozef Vilcek created KAFKA-13196:
------------------------------------
Summary: MirrorMaker 2 not always starting tasks after upgrade
2.4.0 -> 2.7.1
Key: KAFKA-13196
URL: https://issues.apache.org/jira/browse/KAFKA-13196
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 2.7.1
Reporter: Jozef Vilcek
I am using MirrorMaker 2.0 and running it via [
MirrorMaker.java|https://github.com/apache/kafka/blob/2.7/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java]
class. This method will start up `DistributedHerder` and will use
non-functional `advertisedUrl`, and therefore workers can not talk to each
other and coordinate.
After upgrading from version `2.4.0` to `2.7.1` I did notice that when I am
starting up mirror maker ti does not always start tasks - just connector is
executing. Doing some amount of stop and start it will eventually start tasks
too.
After a bit of digging I did noticed that in attempt to configure connector's
task, code ends up in this [1] branch, where configure request is being
forwarded to the leader. From some reason, task configuration is not done on
leader. However, MirrorMaker does not pack RestServer and therefore that
request will never succeed.
I am not sure what is going no or why it does seem to work better on 2.4.0. I
noticed that connector start procedure did involve less callback on 2.4.0 in
connector start sequence [2].
I believe the root cause is change in startup procedure of connectors in
general. In vestion 2.4, when connector is started by the leader here [1] it
will immediately setup and submit configuration for connector's tasks. However,
in 2.7.1, is it more asynchronous. Connector is started here [2]
Does this look like a bug? Or am I doing something wrong?
[1]
https://github.com/apache/kafka/blob/2.7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1494
[2]
https://github.com/apache/kafka/blob/2.4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1236
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
