[jira] [Commented] (KAFKA-7314) MirrorMaker example in documentation does not work
[
https://issues.apache.org/jira/browse/KAFKA-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596995#comment-16596995
]
Jordan Moore commented on KAFKA-7314:
-
The MirrorMaker documentation could be expanded on a bit, but I have also
tested it with two instances, and it works as expected.
What is this port for though?
{code}
-p 9094:9094 \
{code}
And this isn't necessary either, and might be the source of the issue
{code}
-h `hostname` \
{code}
You're only running a single Kafka instance using those commands and those
guides. Please show your config files.
Also DNS shouldn't work from your host to the Docker network. You also don't
need to edit any /etc/hosts file either. If you need to expose the Kafka
advertised listener outside of the Docker network, you need to register them as
such.
{code}
-p 29092:29092 \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP="
PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT" \
-e
KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092"
{code}
The Confluent Quickstart assumes all Kafka Clients (including MirrorMaker) will
be *within* the Docker network.
If you try to run MirrorMaker outside of Docker with the above environment, you
will now need to use {{localhost:29092}} to connect to this broker. You need to
run a second Kafka container and change the port to {{29093}}, for example,
then connect mirror the two like that.
> MirrorMaker example in documentation does not work
> --
>
> Key: KAFKA-7314
> URL: https://issues.apache.org/jira/browse/KAFKA-7314
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Reporter: John Wilkinson
>Priority: Critical
>
> Kafka MirrorMaker as described in the documentation
> [here|https://kafka.apache.org/documentation/#basic_ops_mirror_maker] does
> not work. Instead of pulling messages from the consumer-defined
> {{bootstrap.servers}} and pushing to the producer-defined
> {{bootstrap.servers}}, it consumes and producers on the same topic on the
> same host repeatedly.
> To replicate, set up two instances of kafka following
> [this|https://docs.confluent.io/current/installation/docker/docs/installation/recipes/single-node-client.html]
> guide. The schema registry and rest proxy are unnecessary.
> [Here|https://hub.docker.com/r/confluentinc/cp-kafka/] is the DockerHub page
> for the image. The Kafka version is 2.0.0.
> Using those two kafka instances, go {{docker exec}} into one and set up the
> {{consumer.properties}} and the {{producer.properties}} following the
> MirrorMaker guide.
> Oddly, if you put in garbage unresolvable server addresses in the config,
> there will be an error, despite the configs not getting used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7314) MirrorMaker example in documentation does not work
[
https://issues.apache.org/jira/browse/KAFKA-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590472#comment-16590472
]
Manikumar commented on KAFKA-7314:
--
As I mentioned above, this looks like Docker networking/DNS issue. looks like
you are configuring same hostname ("kafka") for both the hosts. So if we use
"kafka" hostname in both consumer/producer properties, they will always resolve
to single host . Try configuring different hostnames and verify.
> MirrorMaker example in documentation does not work
> --
>
> Key: KAFKA-7314
> URL: https://issues.apache.org/jira/browse/KAFKA-7314
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Reporter: John Wilkinson
>Priority: Critical
>
> Kafka MirrorMaker as described in the documentation
> [here|https://kafka.apache.org/documentation/#basic_ops_mirror_maker] does
> not work. Instead of pulling messages from the consumer-defined
> {{bootstrap.servers}} and pushing to the producer-defined
> {{bootstrap.servers}}, it consumes and producers on the same topic on the
> same host repeatedly.
> To replicate, set up two instances of kafka following
> [this|https://docs.confluent.io/current/installation/docker/docs/installation/recipes/single-node-client.html]
> guide. The schema registry and rest proxy are unnecessary.
> [Here|https://hub.docker.com/r/confluentinc/cp-kafka/] is the DockerHub page
> for the image. The Kafka version is 2.0.0.
> Using those two kafka instances, go {{docker exec}} into one and set up the
> {{consumer.properties}} and the {{producer.properties}} following the
> MirrorMaker guide.
> Oddly, if you put in garbage unresolvable server addresses in the config,
> there will be an error, despite the configs not getting used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7314) MirrorMaker example in documentation does not work
[
https://issues.apache.org/jira/browse/KAFKA-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590425#comment-16590425
]
John Wilkinson commented on KAFKA-7314:
---
Here are the docker commands I am using to start the kafka and zk instances:
{{docker run -d \
--net=confluent \
--name=zookeeper \
-e ZOOKEEPER_CLIENT_PORT=2181 \
confluentinc/cp-zookeeper:latest}}
{{docker run -d \
--net=confluent \
--name=kafka \
-h `hostname` \
-p 9092:9092 \
-p 9094:9094 \
-e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
confluentinc/cp-kafka:latest}}
What I mean when I say the value of {{bootstrap.servers}} doesn't matter is
that regardless of what you put in them, mirrormaker consumes and publishes to
localhost. However, if you put unresolvable garbage in those values,
mirrormaker will fail to start with a DNS resolution error. So mirrormaker
appears to be reading those values, but then not using them.
This is easily 100% reproducible, just launch those two nodes in the docker
containers on two different hosts and try to use mirrormaker on them.
> MirrorMaker example in documentation does not work
> --
>
> Key: KAFKA-7314
> URL: https://issues.apache.org/jira/browse/KAFKA-7314
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Reporter: John Wilkinson
>Priority: Critical
>
> Kafka MirrorMaker as described in the documentation
> [here|https://kafka.apache.org/documentation/#basic_ops_mirror_maker] does
> not work. Instead of pulling messages from the consumer-defined
> {{bootstrap.servers}} and pushing to the producer-defined
> {{bootstrap.servers}}, it consumes and producers on the same topic on the
> same host repeatedly.
> To replicate, set up two instances of kafka following
> [this|https://docs.confluent.io/current/installation/docker/docs/installation/recipes/single-node-client.html]
> guide. The schema registry and rest proxy are unnecessary.
> [Here|https://hub.docker.com/r/confluentinc/cp-kafka/] is the DockerHub page
> for the image. The Kafka version is 2.0.0.
> Using those two kafka instances, go {{docker exec}} into one and set up the
> {{consumer.properties}} and the {{producer.properties}} following the
> MirrorMaker guide.
> Oddly, if you put in garbage unresolvable server addresses in the config,
> there will be an error, despite the configs not getting used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7314) MirrorMaker example in documentation does not work
[
https://issues.apache.org/jira/browse/KAFKA-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590342#comment-16590342
]
Manikumar commented on KAFKA-7314:
--
We have unit/system tests to verify the basic mirrormaker scenarios. A quick
test (start two single node instances and test mirroring) works for me.
I am not sure what do you mean by "the value of bootstrap.servers irrelevant as
long as they are valid when resolving DNS".
Of course, bootstrap.servers names should resolve to valid kafka instance,
otherwise mirrormaker fails while consuming/producing data.
Issue maybe related to docker environment. we need to make sure hostnames are
uniquely resolves to valid kafka instances.
> MirrorMaker example in documentation does not work
> --
>
> Key: KAFKA-7314
> URL: https://issues.apache.org/jira/browse/KAFKA-7314
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Reporter: John Wilkinson
>Priority: Critical
>
> Kafka MirrorMaker as described in the documentation
> [here|https://kafka.apache.org/documentation/#basic_ops_mirror_maker] does
> not work. Instead of pulling messages from the consumer-defined
> {{bootstrap.servers}} and pushing to the producer-defined
> {{bootstrap.servers}}, it consumes and producers on the same topic on the
> same host repeatedly.
> To replicate, set up two instances of kafka following
> [this|https://docs.confluent.io/current/installation/docker/docs/installation/recipes/single-node-client.html]
> guide. The schema registry and rest proxy are unnecessary.
> [Here|https://hub.docker.com/r/confluentinc/cp-kafka/] is the DockerHub page
> for the image. The Kafka version is 2.0.0.
> Using those two kafka instances, go {{docker exec}} into one and set up the
> {{consumer.properties}} and the {{producer.properties}} following the
> MirrorMaker guide.
> Oddly, if you put in garbage unresolvable server addresses in the config,
> there will be an error, despite the configs not getting used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7314) MirrorMaker example in documentation does not work
[
https://issues.apache.org/jira/browse/KAFKA-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589396#comment-16589396
]
John Wilkinson commented on KAFKA-7314:
---
[~omkreddy] As described above, the value of {{bootstrap.servers}} irrelevant
as long as they are valid when resolving DNS. If you have gone through the
example in the docs using two different servers and it works for you, please
post your config and the fact that you could not reproduce.
> MirrorMaker example in documentation does not work
> --
>
> Key: KAFKA-7314
> URL: https://issues.apache.org/jira/browse/KAFKA-7314
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Reporter: John Wilkinson
>Priority: Critical
>
> Kafka MirrorMaker as described in the documentation
> [here|https://kafka.apache.org/documentation/#basic_ops_mirror_maker] does
> not work. Instead of pulling messages from the consumer-defined
> {{bootstrap.servers}} and pushing to the producer-defined
> {{bootstrap.servers}}, it consumes and producers on the same topic on the
> same host repeatedly.
> To replicate, set up two instances of kafka following
> [this|https://docs.confluent.io/current/installation/docker/docs/installation/recipes/single-node-client.html]
> guide. The schema registry and rest proxy are unnecessary.
> [Here|https://hub.docker.com/r/confluentinc/cp-kafka/] is the DockerHub page
> for the image. The Kafka version is 2.0.0.
> Using those two kafka instances, go {{docker exec}} into one and set up the
> {{consumer.properties}} and the {{producer.properties}} following the
> MirrorMaker guide.
> Oddly, if you put in garbage unresolvable server addresses in the config,
> there will be an error, despite the configs not getting used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7314) MirrorMaker example in documentation does not work
[
https://issues.apache.org/jira/browse/KAFKA-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589238#comment-16589238
]
Manikumar commented on KAFKA-7314:
--
looks like both consumer/producer properties are pointing to same cluster.
Can you post your consumer.properties and the producer.properties values and
mirror maker logs?
> MirrorMaker example in documentation does not work
> --
>
> Key: KAFKA-7314
> URL: https://issues.apache.org/jira/browse/KAFKA-7314
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Reporter: John Wilkinson
>Priority: Critical
>
> Kafka MirrorMaker as described in the documentation
> [here|https://kafka.apache.org/documentation/#basic_ops_mirror_maker] does
> not work. Instead of pulling messages from the consumer-defined
> {{bootstrap.servers}} and pushing to the producer-defined
> {{bootstrap.servers}}, it consumes and producers on the same topic on the
> same host repeatedly.
> To replicate, set up two instances of kafka following
> [this|https://docs.confluent.io/current/installation/docker/docs/installation/recipes/single-node-client.html]
> guide. The schema registry and rest proxy are unnecessary.
> [Here|https://hub.docker.com/r/confluentinc/cp-kafka/] is the DockerHub page
> for the image. The Kafka version is 2.0.0.
> Using those two kafka instances, go {{docker exec}} into one and set up the
> {{consumer.properties}} and the {{producer.properties}} following the
> MirrorMaker guide.
> Oddly, if you put in garbage unresolvable server addresses in the config,
> there will be an error, despite the configs not getting used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
