[ 
https://issues.apache.org/jira/browse/KAFKA-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739868#comment-17739868
 ] 

Edoardo Comar edited comment on KAFKA-15144 at 7/4/23 9:50 AM:
---------------------------------------------------------------

Experimenting a bit, setting
{color:#000000}offset.lag.max={color}{color:#a31515}1{color}
and then running a console producer and consumer on the source,
I still see puzzling checkpoints despite what looks a correct offset sync :
 
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=0, 
downstreamOffset=0}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=2, 
downstreamOffset=2}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=4, 
downstreamOffset=4}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=6, 
downstreamOffset=6}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=8, 
downstreamOffset=8}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=10, 
downstreamOffset=10}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=12, 
downstreamOffset=12}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=14, 
downstreamOffset=14}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=16, 
downstreamOffset=16}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=18, 
downstreamOffset=18}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=20, 
downstreamOffset=20}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=22, 
downstreamOffset=22}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=24, 
downstreamOffset=24}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=26, 
downstreamOffset=26}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=28, 
downstreamOffset=28}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=30, 
downstreamOffset=30}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=32, 
downstreamOffset=32}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=34, 
downstreamOffset=34}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=36, 
downstreamOffset=36}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=38, 
downstreamOffset=38}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=40, 
downstreamOffset=40}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=42, 
downstreamOffset=42}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=44, 
downstreamOffset=44}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=46, 
downstreamOffset=46}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=48, 
downstreamOffset=48}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=50, 
downstreamOffset=50}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=52, 
downstreamOffset=52}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=54, 
downstreamOffset=54}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=56, 
downstreamOffset=56}
 
Checkpoint\{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=1, 
downstreamOffset=1, metadata=}
Checkpoint\{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=2, 
downstreamOffset=2, metadata=}
Checkpoint\{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=3, 
downstreamOffset=3, metadata=}
Checkpoint\{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=4, 
downstreamOffset=4, metadata=}
Checkpoint\{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=5, 
downstreamOffset=5, metadata=}
Checkpoint{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, {*}upstreamOffset=30, 
downstreamOffset=29{*}, metadata=}

Checkpoint{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, *upstreamOffset=52, 
downstreamOffset=51,* metadata=}

Checkpoint\{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=57, 
downstreamOffset=57, metadata=}

 
I'd expect the upstreamOffset to be always the same as the upstreamOffset
there are no transactions, and the topic has been created fresh with MM2 running


was (Author: ecomar):
Experimenting a bit, setting
{color:#000000}offset.lag.max={color}{color:#a31515}1{color}
and then running a console producer and consumer on the source,
I still see puzzling checkpoints despite what looks a correct offset sync :
 
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=0, 
downstreamOffset=0}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=2, 
downstreamOffset=2}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=4, 
downstreamOffset=4}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=6, 
downstreamOffset=6}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=8, 
downstreamOffset=8}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=10, 
downstreamOffset=10}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=12, 
downstreamOffset=12}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=14, 
downstreamOffset=14}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=16, 
downstreamOffset=16}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=18, 
downstreamOffset=18}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=20, 
downstreamOffset=20}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=22, 
downstreamOffset=22}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=24, 
downstreamOffset=24}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=26, 
downstreamOffset=26}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=28, 
downstreamOffset=28}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=30, 
downstreamOffset=30}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=32, 
downstreamOffset=32}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=34, 
downstreamOffset=34}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=36, 
downstreamOffset=36}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=38, 
downstreamOffset=38}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=40, 
downstreamOffset=40}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=42, 
downstreamOffset=42}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=44, 
downstreamOffset=44}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=46, 
downstreamOffset=46}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=48, 
downstreamOffset=48}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=50, 
downstreamOffset=50}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=52, 
downstreamOffset=52}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=54, 
downstreamOffset=54}
OffsetSync\{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=56, 
downstreamOffset=56}
 
Checkpoint\{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=1, 
downstreamOffset=1, metadata=}
Checkpoint\{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=2, 
downstreamOffset=2, metadata=}
Checkpoint\{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=3, 
downstreamOffset=3, metadata=}
Checkpoint\{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=4, 
downstreamOffset=4, metadata=}
Checkpoint\{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=5, 
downstreamOffset=5, metadata=}
Checkpoint{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, *upstreamOffset=30, 
downstreamOffset=29,* metadata=}
Checkpoint{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, *upstreamOffset=52, 
downstreamOffset=51,* metadata=}
Checkpoint{consumerGroupId=edogroup, 
topicPartition=source.vf-mirroring-test-edo-0, *upstreamOffset=57, 
downstreamOffset=57,* metadata=}
 
I'd expect the upstreamOffset to be always the same as the upstreamOffset
there are no transactions, and the topic has been created fresh with MM2 running

> MM2 Checkpoint downstreamOffset stuck to 1
> ------------------------------------------
>
>                 Key: KAFKA-15144
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15144
>             Project: Kafka
>          Issue Type: Bug
>          Components: mirrormaker
>            Reporter: Edoardo Comar
>            Assignee: Edoardo Comar
>            Priority: Major
>         Attachments: edo-connect-mirror-maker-sourcetarget.properties
>
>
> Steps to reproduce :
> 1.Start the source cluster
> 2.Start the target cluster
> 3.Start connect-mirror-maker.sh using a config like the attached
> 4.Create a topic in source cluster
> 5.produce a few messages
> 6.consume them all with autocommit enabled
>  
> 7. then dump the Checkpoint topic content e.g.
> {{% bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 --topic 
> source.checkpoints.internal --from-beginning --formatter 
> org.apache.kafka.connect.mirror.formatters.CheckpointFormatter}}
> {{{}Checkpoint{consumerGroupId=edogroup, 
> topicPartition=source.vf-mirroring-test-edo-0, upstreamOffset=3, 
> {*}downstreamOffset=1{*}, metadata={}}}}
>  
> the downstreamOffset remains at 1, while, in a fresh cluster pair like with 
> the source topic created while MM2 is running, 
> I'd expect the downstreamOffset to match the upstreamOffset.
> Note that dumping the offset sync topic, shows matching initial offsets
> {{% bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 --topic 
> mm2-offset-syncs.source.internal --from-beginning --formatter 
> org.apache.kafka.connect.mirror.formatters.OffsetSyncFormatter}}
> {{{}OffsetSync{topicPartition=vf-mirroring-test-edo-0, upstreamOffset=0, 
> downstreamOffset=0{}}}}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to