[jira] [Updated] (KAFKA-7629) Mirror maker goes into infinite loop

2018-11-14 Thread Darshan Mehta (JIRA) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Darshan Mehta updated KAFKA-7629:
-
Description: 
*Setup:*

I have 2 kafka images running Spoify Kafka image : 
[https://hub.docker.com/r/spotify/kafka/|https://hub.docker.com/r/spotify/kafka/)]

Config:

Image 1:
 * host: kafka1
 * zk port : 2181
 * broker port : 9092

Image 2:
 * host: kafka2
 * zk port : 1181
 * broker port : 8092

Producer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka2:8092
{code}
Consumer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true
{code}
 

*Steps to replicate :*
 # Start mirror maker with following command : 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
 --consumer.config  
--num.streams 1 --whitelist topic-1
{code}

 # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
kafka2:8092 --topic topic-1
{code}

 # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
console in Step 2
 # Stop mirror maker with ctrl+C (started in step 1)
 # Restart mirror maker with same command
 # Produce an event onto the same topic (i.e. repeat step 3)
 # Both source and destination will be flooded with the same messages until 
mirror maker is stopped

Surprisingly, source kafka also gets flooded with the same message. I believe 
when restarted, the mirror maker is unable to read the state?

  was:
*Setup:*

I have 2 kafka images running (Spoify Kafka image : 
[https://hub.docker.com/r/spotify/kafka/|https://hub.docker.com/r/spotify/kafka/)])

Config:

Image 1:
 * host: kafka1
 * zk port : 2181
 * broker port : 9092

Image 2:
 * host: kafka2
 * zk port : 1181
 * broker port : 8092

Producer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka2:8092
{code}
Consumer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true
{code}
 

*Steps to replicate :*
 # Start mirror maker with following command : 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
 --consumer.config  
--num.streams 1 --whitelist topic-1
{code}

 # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
kafka2:8092 --topic topic-1
{code}

 # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
console in Step 2
 # Stop mirror maker with ctrl+C (started in step 1)
 # Restart mirror maker with same command
 # Produce an event onto the same topic (i.e. repeat step 3)
 # Both source and destination will be flooded with the same messages until 
mirror maker is stopped

Surprisingly, source kafka also gets flooded with the same message. I believe 
when restarted, the mirror maker is unable to read the state?


> Mirror maker goes into infinite loop
> 
>
> Key: KAFKA-7629
> URL: https://issues.apache.org/jira/browse/KAFKA-7629
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.0.0
> Environment: local
>Reporter: Darshan Mehta
>Priority: Major
>
> *Setup:*
> I have 2 kafka images running Spoify Kafka image : 
> [https://hub.docker.com/r/spotify/kafka/|https://hub.docker.com/r/spotify/kafka/)]
> Config:
> Image 1:
>  * host: kafka1
>  * zk port : 2181
>  * broker port : 9092
> Image 2:
>  * host: kafka2
>  * zk port : 1181
>  * broker port : 8092
> Producer Properties for Mirror maker: 
> {code:java}
> bootstrap.servers=kafka2:8092
> {code}
> Consumer Properties for Mirror maker: 
> {code:java}
> bootstrap.servers=kafka1:9092
> group.id=test-consumer-group
> exclude.internal.topics=true
> {code}
>  
> *Steps to replicate :*
>  # Start mirror maker with following command : 
> {code:java}
> $KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
>  --consumer.config  
> --num.streams 1 --whitelist topic-1
> {code}
>  # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
> {code:java}
> $KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
> kafka2:8092 --topic topic-1
> {code}
>  # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
> console in Step 2
>  # Stop mirror maker with ctrl+C (started in step 1)
>  # Restart mirror maker with same command
>  # Produce an event onto the same topic (i.e. repeat step 3)
>  # Both source and destination will be flooded with the same messages until 
> mirror maker is stopped
> Surprisingly, source kafka also gets flooded with the same message. I believe 
> when restarted, the mirror maker is unable to read the

[jira] [Updated] (KAFKA-7629) Mirror maker goes into infinite loop

2018-11-14 Thread Darshan Mehta (JIRA) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Darshan Mehta updated KAFKA-7629:
-
Description: 
*Setup:*

I have 2 kafka images running (Spoify Kafka image : 
[https://hub.docker.com/r/spotify/kafka/|https://hub.docker.com/r/spotify/kafka/)])

Config:

Image 1:
 * host: kafka1
 * zk port : 2181
 * broker port : 9092

Image 2:
 * host: kafka2
 * zk port : 1181
 * broker port : 8092

Producer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka2:8092
{code}
Consumer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true
{code}
 

*Steps to replicate :*
 # Start mirror maker with following command : 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
 --consumer.config  
--num.streams 1 --whitelist topic-1
{code}

 # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
kafka2:8092 --topic topic-1
{code}

 # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
console in Step 2
 # Stop mirror maker with ctrl+C (started in step 1)
 # Restart mirror maker with same command
 # Produce an event onto the same topic (i.e. repeat step 3)
 # Both source and destination will be flooded with the same messages until 
mirror maker is stopped

Surprisingly, source kafka also gets flooded with the same message. I believe 
when restarted, the mirror maker is unable to read the state?

  was:
*Setup:*

I have 2 kafka images running (Spoify Kafka image : 
[https://hub.docker.com/r/spotify/kafka/)]

Config:

Image 1:
 * host: kafka1
 * zk port : 2181
 * broker port : 9092

Image 2:
 * host: kafka2
 * zk port : 1181
 * broker port : 8092

Producer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka2:8092
{code}
Consumer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true
{code}
 

*Steps to replicate :*
 # Start mirror maker with following command : 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
 --consumer.config  
--num.streams 1 --whitelist topic-1
{code}

 # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
kafka2:8092 --topic topic-1
{code}

 # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
console in Step 2
 # Stop mirror maker with ctrl+C (started in step 1)
 # Restart mirror maker with same command
 # Produce an event onto the same topic (i.e. repeat step 3)
 # Both source and destination will be flooded with the same messages until 
mirror maker is stopped

Surprisingly, source kafka also gets flooded with the same message. I believe 
when restarted, the mirror maker is unable to read the state?


> Mirror maker goes into infinite loop
> 
>
> Key: KAFKA-7629
> URL: https://issues.apache.org/jira/browse/KAFKA-7629
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.0.0
> Environment: local
>Reporter: Darshan Mehta
>Priority: Major
>
> *Setup:*
> I have 2 kafka images running (Spoify Kafka image : 
> [https://hub.docker.com/r/spotify/kafka/|https://hub.docker.com/r/spotify/kafka/)])
> Config:
> Image 1:
>  * host: kafka1
>  * zk port : 2181
>  * broker port : 9092
> Image 2:
>  * host: kafka2
>  * zk port : 1181
>  * broker port : 8092
> Producer Properties for Mirror maker: 
> {code:java}
> bootstrap.servers=kafka2:8092
> {code}
> Consumer Properties for Mirror maker: 
> {code:java}
> bootstrap.servers=kafka1:9092
> group.id=test-consumer-group
> exclude.internal.topics=true
> {code}
>  
> *Steps to replicate :*
>  # Start mirror maker with following command : 
> {code:java}
> $KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
>  --consumer.config  
> --num.streams 1 --whitelist topic-1
> {code}
>  # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
> {code:java}
> $KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
> kafka2:8092 --topic topic-1
> {code}
>  # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
> console in Step 2
>  # Stop mirror maker with ctrl+C (started in step 1)
>  # Restart mirror maker with same command
>  # Produce an event onto the same topic (i.e. repeat step 3)
>  # Both source and destination will be flooded with the same messages until 
> mirror maker is stopped
> Surprisingly, source kafka also gets flooded with the same message. I believe 
> when restarted, the mirror maker is unable to read the state?



--
This message was sent by

[jira] [Updated] (KAFKA-7629) Mirror maker goes into infinite loop

2018-11-14 Thread Darshan Mehta (JIRA) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Darshan Mehta updated KAFKA-7629:
-
Description: 
*Setup:*

I have 2 kafka images running Spoify Kafka image 
[https://hub.docker.com/r/spotify/kafka]

Config:

Image 1:
 * host: kafka1
 * zk port : 2181
 * broker port : 9092

Image 2:
 * host: kafka2
 * zk port : 1181
 * broker port : 8092

Producer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka2:8092
{code}
Consumer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true
{code}
 

*Steps to replicate :*
 # Start mirror maker with following command : 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
 --consumer.config  
--num.streams 1 --whitelist topic-1
{code}

 # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
kafka2:8092 --topic topic-1
{code}

 # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
console in Step 2
 # Stop mirror maker with ctrl+C (started in step 1)
 # Restart mirror maker with same command
 # Produce an event onto the same topic (i.e. repeat step 3)
 # Both source and destination will be flooded with the same messages until 
mirror maker is stopped

Surprisingly, source kafka also gets flooded with the same message. I believe 
when restarted, the mirror maker is unable to read the state?

  was:
*Setup:*

I have 2 kafka images running Spoify Kafka image : 
[https://hub.docker.com/r/spotify/kafka/|https://hub.docker.com/r/spotify/kafka/)]

Config:

Image 1:
 * host: kafka1
 * zk port : 2181
 * broker port : 9092

Image 2:
 * host: kafka2
 * zk port : 1181
 * broker port : 8092

Producer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka2:8092
{code}
Consumer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true
{code}
 

*Steps to replicate :*
 # Start mirror maker with following command : 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
 --consumer.config  
--num.streams 1 --whitelist topic-1
{code}

 # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
kafka2:8092 --topic topic-1
{code}

 # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
console in Step 2
 # Stop mirror maker with ctrl+C (started in step 1)
 # Restart mirror maker with same command
 # Produce an event onto the same topic (i.e. repeat step 3)
 # Both source and destination will be flooded with the same messages until 
mirror maker is stopped

Surprisingly, source kafka also gets flooded with the same message. I believe 
when restarted, the mirror maker is unable to read the state?


> Mirror maker goes into infinite loop
> 
>
> Key: KAFKA-7629
> URL: https://issues.apache.org/jira/browse/KAFKA-7629
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.0.0
> Environment: local
>Reporter: Darshan Mehta
>Priority: Major
>
> *Setup:*
> I have 2 kafka images running Spoify Kafka image 
> [https://hub.docker.com/r/spotify/kafka]
> Config:
> Image 1:
>  * host: kafka1
>  * zk port : 2181
>  * broker port : 9092
> Image 2:
>  * host: kafka2
>  * zk port : 1181
>  * broker port : 8092
> Producer Properties for Mirror maker: 
> {code:java}
> bootstrap.servers=kafka2:8092
> {code}
> Consumer Properties for Mirror maker: 
> {code:java}
> bootstrap.servers=kafka1:9092
> group.id=test-consumer-group
> exclude.internal.topics=true
> {code}
>  
> *Steps to replicate :*
>  # Start mirror maker with following command : 
> {code:java}
> $KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
>  --consumer.config  
> --num.streams 1 --whitelist topic-1
> {code}
>  # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
> {code:java}
> $KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
> kafka2:8092 --topic topic-1
> {code}
>  # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
> console in Step 2
>  # Stop mirror maker with ctrl+C (started in step 1)
>  # Restart mirror maker with same command
>  # Produce an event onto the same topic (i.e. repeat step 3)
>  # Both source and destination will be flooded with the same messages until 
> mirror maker is stopped
> Surprisingly, source kafka also gets flooded with the same message. I believe 
> when restarted, the mirror maker is unable to read the state?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KAFKA-7629) Mirror maker goes into infinite loop

2018-11-14 Thread Darshan Mehta (JIRA) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Darshan Mehta updated KAFKA-7629:
-
Description: 
*Setup:*

I have 2 kafka images running (Spoify Kafka image : 
[https://hub.docker.com/r/spotify/kafka/)]

Config:

Image 1:
 * host: kafka1
 * zk port : 2181
 * broker port : 9092

Image 2:
 * host: kafka2
 * zk port : 1181
 * broker port : 8092

Producer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka2:8092
{code}
Consumer Properties for Mirror maker: 
{code:java}
bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true
{code}
 

*Steps to replicate :*
 # Start mirror maker with following command : 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
 --consumer.config  
--num.streams 1 --whitelist topic-1
{code}

 # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
kafka2:8092 --topic topic-1
{code}

 # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
console in Step 2
 # Stop mirror maker with ctrl+C (started in step 1)
 # Restart mirror maker with same command
 # Produce an event onto the same topic (i.e. repeat step 3)
 # Both source and destination will be flooded with the same messages until 
mirror maker is stopped

Surprisingly, source kafka also gets flooded with the same message. I believe 
when restarted, the mirror maker is unable to read the state?

  was:
*Setup:*

I have 2 kafka images running (Spoify Kafka image : 
[https://hub.docker.com/r/spotify/kafka/)]

Config:

Image 1:
 * host: kafka1
 * zk port : 2181
 * broker port : 9092

Image 2:
 * host: kafka2
 * zk port : 1181
 * broker port : 8092

Producer Properties for Mirror maker:

 
{code:java}
bootstrap.servers=kafka2:8092
{code}
Consumer Properties for Mirror maker:

 

 
{code:java}
bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true
{code}
 

*Steps to replicate :*
 # Start mirror maker with following command : 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
 --consumer.config  
--num.streams 1 --whitelist topic-1
{code}

 # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
kafka2:8092 --topic topic-1
{code}

 # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
console in Step 2
 # Stop mirror maker with ctrl+C (started in step 1)
 # Restart mirror maker with same command
 # Produce an event onto the same topic (i.e. repeat step 3)
 # Both source and destination will be flooded with the same messages until 
mirror maker is stopped

Surprisingly, source kafka also gets flooded with the same message. I believe 
when restarted, the mirror maker is unable to read the state?


> Mirror maker goes into infinite loop
> 
>
> Key: KAFKA-7629
> URL: https://issues.apache.org/jira/browse/KAFKA-7629
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.0.0
> Environment: local
>Reporter: Darshan Mehta
>Priority: Major
>
> *Setup:*
> I have 2 kafka images running (Spoify Kafka image : 
> [https://hub.docker.com/r/spotify/kafka/)]
> Config:
> Image 1:
>  * host: kafka1
>  * zk port : 2181
>  * broker port : 9092
> Image 2:
>  * host: kafka2
>  * zk port : 1181
>  * broker port : 8092
> Producer Properties for Mirror maker: 
> {code:java}
> bootstrap.servers=kafka2:8092
> {code}
> Consumer Properties for Mirror maker: 
> {code:java}
> bootstrap.servers=kafka1:9092
> group.id=test-consumer-group
> exclude.internal.topics=true
> {code}
>  
> *Steps to replicate :*
>  # Start mirror maker with following command : 
> {code:java}
> $KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
>  --consumer.config  
> --num.streams 1 --whitelist topic-1
> {code}
>  # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
> {code:java}
> $KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
> kafka2:8092 --topic topic-1
> {code}
>  # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
> console in Step 2
>  # Stop mirror maker with ctrl+C (started in step 1)
>  # Restart mirror maker with same command
>  # Produce an event onto the same topic (i.e. repeat step 3)
>  # Both source and destination will be flooded with the same messages until 
> mirror maker is stopped
> Surprisingly, source kafka also gets flooded with the same message. I believe 
> when restarted, the mirror maker is unable to read the state?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7629) Mirror maker goes into infinite loop

2018-11-14 Thread Darshan Mehta (JIRA) 
Darshan Mehta created KAFKA-7629:


 Summary: Mirror maker goes into infinite loop
 Key: KAFKA-7629
 URL: https://issues.apache.org/jira/browse/KAFKA-7629
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 2.0.0
 Environment: local
Reporter: Darshan Mehta


*Setup:*

I have 2 kafka images running (Spoify Kafka image : 
[https://hub.docker.com/r/spotify/kafka/)]

Config:

Image 1:
 * host: kafka1
 * zk port : 2181
 * broker port : 9092

Image 2:
 * host: kafka2
 * zk port : 1181
 * broker port : 8092

Producer Properties for Mirror maker:

 
{code:java}
bootstrap.servers=kafka2:8092
{code}
Consumer Properties for Mirror maker:

 

 
{code:java}
bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true
{code}
 

*Steps to replicate :*
 # Start mirror maker with following command : 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config 
 --consumer.config  
--num.streams 1 --whitelist topic-1
{code}

 # Start local kafka console consumer to listen to topic-1 for kafka2:8092 
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server 
kafka2:8092 --topic topic-1
{code}

 # Produce an event to kafka1:9092 - topic-1  -> It will be printed on the 
console in Step 2
 # Stop mirror maker with ctrl+C (started in step 1)
 # Restart mirror maker with same command
 # Produce an event onto the same topic (i.e. repeat step 3)
 # Both source and destination will be flooded with the same messages until 
mirror maker is stopped

Surprisingly, source kafka also gets flooded with the same message. I believe 
when restarted, the mirror maker is unable to read the state?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)