Hi, I am new to MirrorMaker and I have been trying to understand its behavior, in particular about its consumer offset synchronization.
I saw KIP-545 https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0, which talks about this feature implemented in 2.7.0. I thought this feature would allow a consumer to be able to switch from the source cluster and to the mirrored cluster to continue consuming records. I tried to test this feature with 2.7.0 but was unsuccessful. So, I looked at testOneWayReplicationWithAutoOffsetSync of MirrorConnectorsIntegrationTest which seems to test this offset sychronization feature. https://github.com/apache/kafka/blob/trunk/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/MirrorConnectorsIntegrationTest.java#L453 But I see that in this test, the consumer is switched to the new cluster when its offset is already set at "latest" and the assertion is only verifying the consumer to be at the latest offset (i.e., no records are consumed) at the new cluster when it is switched. https://github.com/apache/kafka/blob/trunk/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/MirrorConnectorsIntegrationTest.java#L496 This doesn't distinguish the cases between whether the offset is synchronized or the consumer started at the new cluster with "latest", no? So I modified this test to make the consumer's offset to have some lag when it is switched so that the consumer can consume those remaining records from the new cluster, as in this change https://github.com/elakito/kafka/commit/9063db0f8c4a53f5d9764612af898981d499a7b7 But this test failed. So, I am confused. Is my test wrong or do I misunderstand what KIP-545 is supposed to provide? Any help to understand this matter would be greatly appreciated. Regards, aki
